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ABSTRACT 


The  volume  of  data  available  to  military  decision  makers  is  vast.  Leaders  need  tools  to 
sort,  analyze,  and  present  information  in  an  effective  manner.  Software  complexity  is 
also  increasing,  with  user  interfaces  becoming  more  intricate  and  interactive.  The  Data 
Visualization  Tool  (DaViTo)  is  an  effort  by  TRAC  Monterey  to  produce  a  tool  for  use  by 
personnel  with  little  statistical  background  to  process  and  display  this  data.  To  meet  the 
program  goals  and  make  analytical  capabilities  more  widely  available,  the  user  interface 
and  data  representation  techniques  need  refinement.  This  usability  test  is  a  task-oriented 
study  using  eye-tracking,  data  representation  techniques,  and  surveys  to  generate 
recommendations  for  software  improvement.  Twenty-four  subjects  participated  in  three 
sessions  using  DaViTo  over  a  three-week  period.  The  first  two  sessions  consisted  of 
training  followed  by  basic  reinforcement  tasks,  evaluation  of  graphical  methods,  and  a 
brief  survey.  The  final  session  was  a  task-oriented  session  followed  by  graphical 
representations  evaluation  and  an  extensive  survey.  Results  from  the  three  sessions  were 
analyzed  and  37  recommendations  generated  for  the  improvement  of  DaViTo. 
Improving  software  latency,  providing  more  graphing  options  and  tools,  and  inclusion  of 
an  effective  training  product  are  examples  of  important  recommendations  that  would 
greatly  improve  usability. 
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I.  INTRODUCTION 


A.  PURPOSE 

In  the  early  years  of  computers  and  the  software  to  use  and  operate  them,  most 
applications  were  developed  by  highly  trained  and  specialized  personnel.  Software  was 
designed  for  a  user  with  commensurate  specialized  skills  in  the  computing  arena.  Jobs 
that  required  the  use  of  computers  also  required  specific  training  or  education  in  order  to 
use  them.  As  computing  power  began  to  increase  and  the  use  of  computers  spread  across 
professions  and  into  homes,  there  were  more  users  with  little  or  no  education  or  training 
in  the  field.  Software  development  evolved  with  this,  merging  toward  a  central  theme  of 
usability  for  the  end  user.  This  enabled  computers  and  software  to  traverse  into  all  fields 
of  work  for  a  full  range  of  users.  Usability  has  many  definitions  but  in  software 
engineering  it  is  “the  quality  of  a  system  with  respect  to  ease  of  learning,  ease  of  use,  and 
user  satisfaction”  (Rosson  &  Carroll,  2002).  These  three  usability  perspectives  are 
directly  reflected  in  a  summary  statement  of  the  goals  TRAC  Monterey  stated  when 
developing  the  Data  Visualization  Tool  (DaViTo).  Their  intention  was  to  develop  an 
open  source  tool  that  would  be  easy  to  use  for  someone  unfamiliar  with  statistical 
methods  to  produce  useful  visualizations  and  analysis  of  spatial-temporal  data,  and  be 
able  to  present  that  output  in  a  way  that  could  be  understood  by  ground  commanders 
(Evangelista,  Henry,  Buttrey,  &  Whitaker,  2012).  The  overall  purpose  of  this  thesis  is  to 
determine  how  DaViTo  in  its  current  state  needs  to  be  improved  to  meet  the  usability 
goals  it  was  originally  developed  to  fulfill. 

B.  OBJECTIVES 

The  goal  is  to  provide  concrete  and  substantive  recommendations  to  improve  the 
usability  and  workflow  of  the  DaViTo  software.  Research  for  this  thesis  began  with  two 
main  objectives,  both  with  subsets  of  secondary  objectives. 

The  primary  objective  consists  of  evaluating  the  usability  and  efficiency  of  the 
DaViTo  to  determine  if  the  workflow  to  complete  a  task  matches  how  a  user  with 
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minimal  training  thinks  the  workflow  should  be  carried  out.  The  following  subset  of 
secondary  objectives  was  used  to  complete  the  first  primary  objective. 

•  Identify  speed  and  accuracy  to  complete  tasks  by  investigating  mouse 
clicking  trends,  task  completion  times,  and  number  of  correct  answers. 

•  Identify  misunderstandings  of  the  software  interface  such  as  button 
symbology,  names,  mouse-overs,  or  locations. 

•  Utilize  eye-tracking  characteristics  such  as  gaze  patterns  and  dwell  times 
to  identify  software  inefficiencies. 

The  second  objective  is  to  identify  areas  where  efficiency  and  usability  can  be 
improved,  and  translate  these  into  actionable  recommendations.  Again,  a  subset  of 
secondary  objectives  was  used  to  complete  the  primary  objective. 

•  Pinpoint  ways  to  improve  the  software’s  visual  palette  in  a  manner  that 
will  increase  infonnation  transfer  to  users. 

•  Identify  button  improvements  such  as  location  and  identification  that  will 
increase  ease  of  use. 

•  Identify  ways  to  optimize  input  device  use  to  increase  efficiency. 

The  final  objective  was  to  determine  if  there  were  deficiencies  in  the  use  of  data 
representation  techniques  within  DaViTo,  and  how  to  correct  them.  The  secondary 
objectives  to  complete  this  were  as  follows. 

•  Identify  graph  types  that  best  represent  the  statistical  analysis  of  typical 
data  sets  used  in  DaViTo. 

•  Determine  additional  graphing  tools  needed  to  improve  output  statistical 
analysis  by  users  with  little  or  no  statistical  training. 

C.  SCOPE  AND  LIMITATIONS 

This  thesis  is  limited  to  the  usability  of  the  DaViTo  user  interface  itself.  There  is 
no  intention  to  investigate  the  software  behind  DaViTo  despite  one  of  the  software  goals 
being  to  allow  user  modification  of  the  code  for  customization  to  individual  needs. 
DaViTo  is  a  broad  program  that  draws  from  three  different  open  source  software 
packages,  R,  JFreeChart,  and  OpenMap.  Additionally,  the  goal  was  to  test  the  usability 
of  new  users  rather  than  analysts,  so  advanced  functionality  of  the  software  such  as 
outputting  data  to  R  for  advanced  statistical  analysis  is  not  evaluated.  Due  to  time 
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constraints  study  participation  is  limited  to  24  subjects  at  approximately  three  hours  each. 
This  sample  size  is  large  enough  to  thoroughly  evaluate  resultant  data  and  sufficient  for 
potential  loss  or  corruption  of  individual  data  sets,  yet  small  enough  to  be  tested  within 
research  lab  and  personnel  time  restrictions. 

Eye-tracking  is  a  key  component  of  the  usability  testing  of  this  thesis,  and  is  not 
known  to  have  been  frequently  used  previously  in  this  area  of  research.  While  the  eye¬ 
tracking  itself  and  the  statistics  associated  with  it  are  of  interest,  the  scope  of  this  thesis 
limits  the  use  of  eye-tracking  results.  They  were  used  solely  for  developing 
recommendations  for  software  improvement  in  the  areas  of  workflow  analysis  and 
usability.  There  is  a  large  amount  of  data  available  from  the  EyeWorks  software  such  as 
blink  data,  saccades,  and  cognitive  load  based  on  pupil  contraction.  Eye-tracking  was 
only  a  component  of  the  data  evaluated  and  as  such  only  scan  patterns  and  fixations  were 
used  in  the  end.  The  outcome  of  the  eye-tracking  performance  is  not  evaluated  outside  of 
these  areas. 

D.  THESIS  ORGANIZATION 

Chapter  I  -  Introduction.  This  chapter  describes  the  purpose  and  overall 
objectives  of  the  thesis,  as  well  as  identifies  the  research  scope  and  limitations. 

Chapter  II  -  Background.  This  chapter  introduces  and  explains  the  DaViTo 
software  and  usability.  Additionally,  the  purpose  of  usability  testing  generally  and  with 
respect  to  DaViTo  is  broken  down. 

Chapter  III  -  Methodology.  The  experimental  design  is  discussed  in  depth,  to 
include  training  and  tasks,  subject  participation,  and  the  actual  experimental  procedure. 

Chapter  IV  -  Results  and  Data  Analysis.  This  chapter  presents  results  from  the 
collected  data  and  explains  how  the  various  sources  are  analyzed.  The  findings  of  the 
data  analysis  are  also  discussed. 

Chapter  V  -  Recommendations.  This  is  the  consummation  of  the  research  into 
recommendations  for  change  and  improvement  of  the  software  to  improve  its  usability. 
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This  is  input  for  TRAC  Monterey  as  the  software  continues  to  be  improved,  and  is  broken 
down  into  four  categories:  recommendations  based  on  task  results,  graph  results,  survey 
results,  and  author  recommendations. 

Chapter  VI  -  Conclusion  and  Future  Work.  The  final  chapter  will  summarize  the 
results  and  recommendations,  and  detail  future  recommended  work. 

Appendix  A  -  Consolidated  Recommendations.  This  appendix  contains  all 
37  recommendations  to  TRAC  Monterey  in  one  table. 

Appendix  B  -  Additional  Task  Results.  This  appendix  contains  additional  results 
from  the  task  analysis  portion. 

Appendix  C  -  Additional  Graph  Results.  This  appendix  contains  additional 
results  from  the  graph  analysis  portion. 

Appendix  D  -  Additional  Survey  Data.  This  appendix  contains  additional  results 
from  the  survey  data.  It  is  divided  into  the  three  common  questions  for  all  three  sessions’ 
data  and  the  final  survey,  and  contains  all  subject  free-form  input. 

Appendix  E  -  Task  Worksheet.  This  appendix  contains  the  task  list  worksheet 
used  by  the  experiment  proctor  for  the  final  session,  which  lists  all  1 1  tasks. 

Appendix  F  -  Approved  Institutional  Review  Board  (IRB)  Documents,  Including 
the  Call  for  Participants,  Consent  Form,  Demographic  Survey,  and  Final  Survey 
questions.  This  appendix  contains  the  approved  Institutional  Review  Board  Protocol 
cover  sheet  and  author-generated  documents  for  the  usability  test. 
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II.  BACKGROUND 


A.  WHAT  IS  DA  VITO 

The  Data  Visualization  Tool  (DaViTo)  was  developed  by  TRAC  Monterey  to 
give  U.S.  Army  personnel  a  data  exploration  tool  that  could  be  used  to  discover  patterns 
and  conduct  analysis  of  the  huge  amounts  of  data  that  are  generated  in  today’s 
sophisticated  military  operational  environments.  From  the  DaViTo  project  report: 

DaViTo  specializes  in  the  exploration  of  point  process  data  that  contains 
both  a  spatial  and  temporal  component.  Pre-loaded  with  simple  yet  novel 
data  visualization  techniques,  DaViTo  can  enhance  the  analyst’s  ability  to 
understand  point  processes  across  time  and  space.  DaViTo  provides  an 
interface  that  allows  users  to  tessellate  geographic  areas,  define  unique 
geographical  areas,  or  load  predefined  geographic  polygons.  Once 
geographic  areas  of  interest  have  been  defined,  DaViTo  provides  a  suite  of 
data  exploration  methods  that  can  be  applied  to  the  data  that  resides  within 
these  areas.  Since  DaViTo  is  government  owned  software  built  solely 
with  open  source  software,  the  potential  of  this  software  is  limited  only  by 
the  skills  and  expertise  of  the  user  and  developer  community. 
(Evangelista,  Henry,  Buttrey,  &  Whitaker,  2012) 

DaViTo  was  developed  with  four  goals  in  mind  for  the  software.  One  was  that  it 
be  capable  of  producing  useful  visualizations  and  analysis  of  data  with  a  spatial-temporal 
component.  This  is  an  essential  part  of  the  program,  as  it  was  desired  to  convert  the  vast 
amounts  of  data  into  something  that  can  be  interpreted.  This  leads  into  a  second  goal, 
representing  the  data  with  statistical  and  graphical  output  that  can  be  grasped  and 
understood  by  ground  commanders.  The  importance  of  this  was  to  allow  them  to  take  the 
data  input  to  the  tool,  transfonn  that  data  into  a  visual  representation  that  shows  the 
points  of  significance,  and  present  it  to  decision-makers  in  a  logical  and  comprehensible 
manner.  A  third  goal  seen  as  very  important  to  TRAC  Monterey  was  to  distribute  the 
software  as  open  source,  created  from  open  source  programs.  The  point  in  this  goal  was 
to  give  users  the  power  to  modify  the  tool  to  meet  their  specific  needs,  providing  an 
instrument  that  can  be  continually  adapted  by  the  development  community.  The  final 
goal  was  that  the  software  would  be  “easy  to  use  for  someone  unfamiliar  with  statistical 
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methods  and  Geographic  Information  Systems  (GIS)”  (Evangelista,  Henry,  Buttrey,  & 
Whitaker,  2012).  This  goal  was  the  driving  force  for  the  conduct  of  this  thesis.  TRAC 
Monterey’s  approach  was  to  conduct  a  usability  test  of  the  software  package,  after  its 
initial  release,  to  give  recommendations  for  improvement  that  could  be  applied  to  a  future 
revision  of  DaViTo  that  would  better  meet  this  goal.  It  was  identified  that  by  making  the 
tool  more  usable  it  could  be  utilized  further,  beyond  the  use  of  an  Army  analyst,  such  as 
in  deployed  environments,  and  potentially  by  other  branches  of  the  anned  forces. 

DaViTo  is  an  open  source  program  that  is  actually  a  compilation  of  three  open 
source  software  packages.  The  open  source  aspect  was  a  goal  not  only  to  allow  for 
customization,  but  also  for  fiscal  reasons.  Similar  commercial  systems  such  as  ArcGIS 
are  very  powerful,  but  widespread  use  is  not  possible  due  to  the  price.  Additionally, 
commercially  available  products  have  the  ability  to  perform  some  of  the  tasks  that 
DaViTo  does;  however,  none  provide  the  merger  of  functionality  that  DaViTo  possesses. 
As  such,  DaViTo  was  developed  using  three  open  source  software  tools. 

The  first,  OpenMap,  developed  by  Raytheon  subsidiary,  BBN  (Bolt,  Beranek,  and 
Newman)  Systems,  is  “a  Java  Beans  based  toolkit  for  building  applications  and  applets 
needing  geographic  information”  (Raytheon  BBN  Technologies,  2005).  OpenMap  is  the 
GIS  portion  of  DaViTo  that  provides  the  mapping  capability.  It  is  the  root  of  the 
interface  for  users,  allowing  interaction  with  the  map  and  display  of  data  on  it.  Using 
OpenMap,  users  have  the  ability  to  load  and  manipulate  various  Shapefiles,  datasets,  and 
maps  in  multiple  formats. 

JFreeChart  was  the  second  open  sources  software  tool  implemented  in  DaViTo. 
JFreeChart  was  developed  by  David  Gilbert  of  Object  Refinery  Fimited,  and  is  a  widely 
used  Java  chart  library.  It  allowed  the  developers  of  DaViTo  to  incorporate  professional 
quality  data  representations  into  the  project.  The  ability  to  produce  high  quality  visual 
output  was  necessary  for  both  display  when  using  the  software  and  for  import  to  planning 
briefs  and  training. 

Statistical  analysis  was  incorporated  into  DaViTo  with  the  use  of  the  R  Project  for 
Statistical  Computing.  This  open  source  software,  known  as  R,  “is  a  language  and 
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environment  for  statistical  computing  and  graphics.  R  provides  a  wide  variety  of 
statistical  and  graphical  techniques,  and  is  highly  extensible”  (The  R  Foundation  for 
Statistical  Computing).  By  integrating  R  into  DaViTo,  the  ability  to  conduct  complex 
statistical  analysis  is  given  to  the  user,  and  is  only  limited  by  their  skill  level. 

B.  WHAT  IS  USABILITY 

When  the  tenn  usability  is  brought  up,  its  actual  meaning  depends  upon  the 
context  within  which  it  is  used.  Common  definitions  from  the  Merriam- Webster 
Dictionary  are  “capable  of  being  used”  or  “convenient  and  practicable  for  use”  (Merriam- 
Webster,  Incorporated,  2012).  In  the  context  of  computer  software  development  and 
testing  it  has  a  much  more  specific  definition.  In  Rosson  and  Carroll’s  Usability 
Engineering,  usability  is  defined  as  “the  quality  of  a  system  with  respect  to  ease  of 
learning,  ease  of  use,  and  user  satisfaction”  (Rosson  &  Carroll,  2002).  It  is  even  more 
generally  described  in  The  Handbook  of  Usability  Testing  as  “what  makes  something 
usable  is  the  absence  of  frustration  in  using  it”  and  further  defined  as  “when  a  product  is 
truly  usable,  the  user  can  do  what  he  or  she  wants  to  do  the  way  he  or  she  expects  to  be 
able  to  do  it,  without  hindrance,  hesitation,  or  questions”  (Rubin  &  Chisnell,  2008).  Both 
of  these  definitions  apply  to  the  purpose  of  this  thesis  research.  With  the  overall  goal  of 
developing  recommendations  for  improvements  to  the  usability  of  the  DaViTo  software, 
clearly  improving  the  ease  of  learning,  ease  of  use,  and  user  satisfaction,  while 
minimizing  user  frustration,  would  be  drivers  for  these  recommendations.  Throughout 
the  development  of  the  experiment  and  subject  trials  frustration  was  frequently  seen  with 
the  software,  pointing  to  areas  where  improving  the  ease  of  use  would  improve 
satisfaction. 

C.  PURPOSE  OF  USABILITY  TESTING 

The  purpose  of  usability  testing  in  the  context  of  this  thesis  is  to  test  the  usability 
performance  and  workflow  functionality  of  the  DaViTo  software.  The  development  of 
this  experiment  was  driven  in  large  part  by  definitions,  designs,  and  procedures  outlined 
in  two  references.  Rosson  and  Carroll  cite  usability  professionals  from  Digital 
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Equipment  Corporation  describing  usability  engineering  as  “using  the  term  to  refer  to 
concepts  and  techniques  for  planning,  achieving,  and  verifying  objectives  for  system 
usability.  The  key  idea  is  that  measurable  usability  goals  must  be  defined  early  in 
software  development,  and  then  assessed  repeatedly  during  development  to  ensure  that 
they  are  achieved”  (Rosson  &  Carroll,  2002).  While  TRAC  Monterey  developed  goals 
for  the  development  of  the  DaViTo  software,  it  did  not  follow  this  definition  with  respect 
to  end  user  usability.  The  Handbook  for  Usability  Testing  refers  to  usability  testing  in 
the  same  manner  in  which  it  was  desired  to  test  DaViTo  in  this  post-development 
method.  The  authors  deem  it  “a  process  that  employs  people  as  testing  participants  who 
are  representative  of  the  target  audience  to  evaluate  the  degree  to  which  a  product  meets 
specific  usability  criteria”  (Rubin  &  Chisnell,  2008).  Again,  throughout  the  development 
of  DaViTo,  no  subject  testing  was  conducted  to  refine  the  software  and  ensure  the  stated 
goal  of  “easy  to  use  for  someone  unfamiliar  with  statistical  methods  and  Geographic 
Information  Systems”  (Evangelista,  Henry,  Buttrey,  &  Whitaker,  2012).  Providing 
recommendations  that  would  aid  TRAC  Monterey  in  meeting  this  goal  in  future  releases 
of  the  DaViTo  product  was  the  purpose  of  this  usability  test. 

As  the  thesis  was  conceived  and  the  experiment  methodology  began  to  be 
formulated,  it  was  necessary  to  break  usability  into  several  attributes,  most  of  which  were 
derived  from  The  Handbook  of  Usability  Testing  (Rubin  &  Chisnell,  2008).  The 
attributes  are  usefulness,  efficiency,  effectiveness,  leamability,  and  satisfaction,  which 
are  described  in  the  matrix  in  Table  1,  and  provided  the  basis  for  the  different  sections  of 
the  surveys  developed  for  the  experiment,  contained  in  Appendix  F.  In  addition  to  the 
five  attributes  identified  by  Rubin  and  Chisnell,  categories  for  improvements  and 
visualizations  were  used  as  well. 
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Attribute 

Description 

Usefulness 

The  degree  to  which  a  product  enables  a  user  to  achieve  his  or  her 
goals,  and  is  an  assessment  of  the  user’s  willingness  to  use  the 
product  at  all. 

Efficiency 

The  quickness  with  which  the  user’s  goal  can  be  accomplished 
accurately  and  completely  and  is  usually  a  measure  of  time. 

Effectiveness 

The  extent  to  which  the  product  behaves  in  the  way  that  users  expect 
it  to  and  the  ease  with  which  users  can  use  it  to  do  what  they  intend. 

Learnability 

A  part  of  effectiveness  and  has  to  do  with  the  user’s  ability  to 
operate  the  system  to  some  defined  level  of  competence  after  some 
predetermined  amount  and  period  of  training. 

Satisfaction 

Refers  to  the  user’s  perceptions,  feelings,  and  opinions  of  the 
product. 

Improvements 

Raw  recommendations  from  the  subjects  with  regard  to  ways  to 
improve  the  software. 

Visualizations 

Refers  to  the  user’s  interpretation  of  the  user  interface  and  used  to 
determine  if  it  is  inherently  usable  in  its  current  state. 

Table  1.  Matrix  of  usability  attributes  and  their  descriptions  (Rubin  &  Chisnell,  2008). 
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III.  METHODOLOGY 


A.  OVERVIEW  AND  OBJECTIVES 

The  overall  objective  of  the  experiment  was  to  collect  objective  and  subjective 
data  from  participants  using  the  DaViTo  software  that  could  then  be  transformed  into 
actionable  recommendations  for  improvements  to  the  DaViTo  software.  Data  collection 
was  conducted  through  a  number  of  different  methods.  The  first  was  through  a 
demographic  survey  contained  in  Appendix  F.  Another  method  was  a  set  of 
questionnaires  at  the  conclusion  of  each  session,  with  the  final  questionnaire  being  very 
extensive,  also  contained  in  Appendix  F.  The  questionnaires  contained  questions  with 
selectable  multiple  choice  responses  as  well  as  fill  in  the  blank  where  subjects  could  give 
free-fomi  opinionated  answers.  Data  was  also  collected  by  evaluating  subject’s  ability  to 
evaluate  graphical  representations  in  an  accurate  and  timely  manner.  The  final  method  of 
data  collection  utilized  automated  eye-tracking  to  collect  mouse  clicks,  reaction  time,  and 
gaze  data  such  as  fixations  and  scan  patterns. 

Following  the  call  for  participants  and  subsequent  scheduling,  the  experiment 
took  place  over  three  sessions.  The  first  two  sessions  consisted  of  a  brief  training  tutorial 
followed  by  some  simple  reinforcement  tasks  and  questions.  The  objective  of  the 
tutorials  was  to  present  a  method  of  software  functionality  to  the  participant  on  one 
screen,  and  for  them  to  follow  along  on  their  own  screen.  The  objectives  of  the  final 
session  were  completion  of  more  complex  tasks,  data  representation  questions,  and  an 
extensive  questionnaire. 

B.  EQUIPMENT 

1.  Hardware 

The  main  hardware  used  for  the  experiment  consisted  of  two  laptops,  one  desktop, 
and  a  set  of  stereo  cameras  with  an  IR  tracking  pod.  Additional  hardware  included 
multiple  displays,  a  keyboard,  and  a  mouse.  For  the  final  session  the  Gateway  laptop  was 
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removed  because  there  was  no  video  tutorial.  The  data  was  kept  securely  on  the  desktop 
computer  then  moved  to  a  secure  folder  on  the  share  drive  for  analysis.  The  desktop  had 
the  following  specifications: 

•  Operating  System:  Windows  7  Professional  (64-bit) 

•  Manufacturer:  Hewlett-Packard  (HP) 

•  Model:  HP  2800  Workstation 

•  Processor:  Intel(R)  Xeon(R)  CPU  X5560  @  2.80GHz 

•  RAM:  6.00GB 

•  Screens:  Two  24  Inch  LCD  Displays 

The  laptop  for  tutorial  playback  had  the  following  specifications: 

•  Operating  System:  Microsoft  Windows  XP  (32-bit) 

•  Manufacturer:  Gateway 

•  Processor:  Mobile  AMD  Sempron™  3300+  1.99  GHz 

•  RAM:  448  MB 

•  DVD  Player:  Philips  DVD+RW  model  SDVD8441 

•  Screen:  16  in  LCD  display 

The  laptop  for  faceLAB  control  had  the  following  specifications: 

•  System:  Microsoft  Windows  XP  (32-bit) 

•  Manufacturer:  Hewlett  Packard 

•  Processor:  Intel(R)  Core™2  CPU  T7200  @  2.00  GHz 

•  RAM:  .99  GB 

•  Screen:  1 5  in  LCD  display 

The  subjects  utilized  the  following  input  devices:  a  Dell  two-button  mouse  with  a 
scrolling  wheel  in  the  center  as  well  as  a  standard  Dell  QWERTY  keyboard. 
Additionally,  directly  in  front  of  the  keyboard  was  a  pair  of  stereo  cameras  and  an  IR  pod 
that  took  input  from  the  subject’s  eyes.  The  IR  pod  and  stereo  camera  set  were 
manufactured  by  Seeing  Machines;  model  number  IRPD-PI-R2. 
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2. 


Software 


Three  software  packages  were  used  for  the  experiment.  The  program  used  to 
initiate  and  control  subject  eye-tracking  was  faceLAB,  version  5.0.5.  The  experiment 
was  designed  and  implemented  using  the  EyeWorks  software  suite,  version  3.7.3.145. 
Finally,  the  DaViTo  software  package  was  used  by  the  participants  and  was  the  software 
of  interest  and  evaluation 

Data  was  collected  automatically  via  the  eye-tracking  software  faceLAB  and 
EyeWorks.  FaceLAB  was  the  software  vehicle  to  track  subject’s  eyes  and  facial  features. 
It  was  coupled  with  the  EyeWorks  software,  which  would  receive  the  automated  data 
from  faceLAB,  and  record  it  for  future  analysis.  The  EyeWorks  suite  was  the  backbone 
of  the  experimental  design,  and  is  an  all-inclusive  research  software  package  for  users  of 
eye-tracking.  The  software  has  three  functional  areas,  Design,  Record,  and  Analyze,  all 
of  which  were  used.  The  Design  component  was  used  to  format  the  experiment  from 
beginning  to  end  for  each  session.  All  tasks,  questions,  and  surveys  were  built  into  the 
Design  shell.  The  Record  component  ran  each  session.  The  script  was  started  at  the 
beginning  of  each  session,  and  within  the  Record  window  all  tasks  were  prompted, 
answers  collected,  and  surveys  conducted.  This  division  of  the  software  package  was 
also  used  to  capture  not  only  eye  movement,  but  launch  the  experiment,  partition  portions 
of  each  session  for  ease  of  data  analysis,  and  record  mouse  clicks.  It  was  also  useful  for 
real  time  observation  of  the  subjects  view  and  where  their  gaze  was  directed  during  the 
experiment,  as  well  as  providing  a  video  recording  of  every  session.  EyeWorks  analyze 
was  then  used  to  provide  heat  maps  and  graphs  of  gaze  data,  display  mouse  clicks,  and 
output  survey  answers  to  Excel. 

3.  Physical  Setup 

The  arrangement  of  the  physical  hardware,  shown  in  Figure  1,  was  identical  for 
all  three  sessions.  The  exception  was  item  3  in  Figure  1,  the  laptop  used  for  the  video 
tutorials  in  sessions  one  and  two,  which  was  removed  prior  to  session  three.  The  layout 
of  all  hardware  components  shown  in  Figure  1  are  described  in  Table  2. 
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Figure  1.  Birds-eye-view  diagram  of  the  equipment  configuration.  Descriptions  of 
each  item  are  in  Table  2. 


Number  in 
Diagram 

Description 

1 

24  Inch  LCD  Screen  -  Displayed  DaViTo  software,  tasks,  and  surveys 
that  were  created  in  Eye  Works. 

2 

Secondary  24  Inch  LCD  Screen  -  Display  for  the  experiment  controller 
to  monitor  EyeWorks  tracking  data. 

3 

Gateway  Laptop  -  Displayed  the  video  tutorials.  This  was  a  standalone 
device  and  was  removed  prior  to  the  final  session. 

4 

Hewlett  Packard  Laptop  -  Controlled  the  faceLAB  software.  This  was 
a  standalone  device  facing  the  experiment  controller. 

5 

Infrared  and  Stereo  Camera  Pod  -  Precision  located  to  track  the 
subject’s  eyes  for  best  results  while  not  obscuring  the  subjects  view. 

6 

Keyboard  -  Input  device  to  respond  to  survey  questions. 

7 

Mouse  -  Input  device  to  allow  the  subject  to  interact  with  the  DaViTo 
and  EyeWorks  software  via  the  HP  desktop. 

Table  2.  Description  of  items  in  equipment  configuration  illustrated  in  Figure  1 . 
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C.  DESIGN  OF  THE  EXPERIMENT 

1.  Training  Tutorial  Development 

In  order  to  standardize  training  for  the  DaViTo  software  and  stay  within  time 
constraints,  the  training  materials  were  designed  as  a  recorded  tutorial.  Microsoft 
PowerPoint  2010,  version  14.0.6123.5001,  was  used  to  generate  slides  showing  the 
subjects  what  to  do.  The  slides  were  then  narrated  and  recorded  using  Camtasia  Studio 
version  7.  The  testing  script  was  developed  using  EyeWorks  Design.  Our  Institutional 
Review  Board  (IRB)  submitted  time  constraint  was  three  one  hour  sessions.  The  goal 
was  to  keep  both  tutorial  sessions  less  than  20  minutes  each  to  allow  ample  time  for  task 
accomplishment,  data  representation  questions,  and  questionnaire  completion.  They  were 
also  designed  with  ample  breaks  and  pause  points  so  subjects  would  not  become  lost  in 
the  tutorials. 

2.  Questionnaire  Development 

The  final  portion  of  each  session  was  a  series  of  questions  drawn  from  the 
extensive  list  of  questions  submitted  with  the  IRB  package  in  Appendix  F.  The 
questionnaire  portion  of  each  session  began  with  the  same  three  questions.  These 
questions  asked  the  subject  to  rate  the  training  for  the  first  two  sessions,  how  much  more 
training  they  think  is  needed  to  use  DaViTo  in  the  field,  and  to  list  their  general  likes  and 
dislikes.  The  final  survey  then  had  a  much  more  extensive  questionnaire  consisting  of 
the  first  three  questions  and  then  approximately  40  more  questions  derived  from  the 
usability  factors  matrix  in  Table  1.  The  questions  were  a  mixture  of  multiple  choice  and 
free  entry.  These  questions  were  designed  to  identify  deficiencies  with  respect  to  the 
usability  attributes,  as  well  as  elicit  recommendations  from  the  participant  that  would  be 
directly  targeted  to  each  attribute. 
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3.  Training  Session  One 


The  first  session  was  limited  to  software  interface  introduction.  The  video  tutorial 
was  exactly  18  minutes  long.  It  began  by  introducing  the  thesis  topic,  the  software 
designers,  and  the  authors.  Next,  the  purpose  of  DaViTo,  its  four  goals,  and  six  main 
functions  were  introduced  to  give  the  participants  some  background  information 
pertaining  to  the  software’s  design  functionality.  Following  the  introduction,  the  user 
interface  was  introduced.  The  user  interface  was  divided  into  five  functional  areas,  as 
shown  in  Figure  2.  Tutorial  session  one  focused  only  on  the  Map  Controls,  Data  Set,  and 
Map  and  Data  Display  Area  portions  of  the  interface,  with  the  Shapefiles  and  Display 
portions  to  be  covered  in  session  two.  To  begin,  the  Data  Set  section  of  the  interface  was 
introduced,  and  the  participant  was  shown  how  to  load  various  data  sets  into  the  software. 
They  were  then  instructed  on  how  to  manipulate  them,  such  as  removing  data  sets  or 
changing  colors.  This  section  was  taught  first  to  facilitate  having  a  data  set  open  in  the 
software  for  the  remainder  of  the  tutorial.  Once  complete,  instruction  moved  to  the  Map 
Controls  portion  of  the  DaViTo  interface.  Here,  the  selectable  drop  down  menus  and 
software  control  buttons  were  each  introduced,  continuing  in  the  tutorial  style  with  the 
participant  following  along  on  their  own  screen.  To  introduce  the  controls  they  were 
each  discussed  and  then  some  were  used  to  manipulate  the  Map  and  Data  Display  Area. 
Pause  points  were  also  inserted,  and  the  participant  was  informed  that  they  could  pause 
the  tutorial  and  take  additional  time  to  explore  the  functionalities  just  discussed  if  they 
desired. 
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Figure  2.  Interface  introduction  slide  from  Tutorial  Session  One.  This  slide  shows  the 
breakdown  of  the  DaViTo  interface  into  five  functional  areas. 


4.  Training  Session  Two 

The  second  session  video  tutorial  began  with  a  brief  review  of  the  initial  session, 
followed  by  more  advanced  instruction.  This  second  tutorial  was  slightly  shorter  at  only 
15  minutes  and  45  seconds.  Following  the  introduction,  a  brief  review  of  the  Map 
Controls,  Data  Set,  and  Map  and  Data  Display  Area  portions  of  the  DaViTo  interface  was 
conducted,  with  an  opportunity  for  the  user  to  pause  the  tutorial  to  interact  and  refresh 
themselves  with  any  portions  of  the  interface  that  they  felt  necessary.  Once  complete,  the 
focus  shifted  to  employing  Shapefiles,  manipulating  statistical  output,  and  conducting 
statistical  analysis,  all  within  the  Map  and  Data  Display  Area.  While  this  required  use  of 
the  previously  covered  Map  Controls  and  Data  Set  areas  of  the  interface,  it  made  frequent 
use  of  the  Shapefiles  and  Display  regions.  The  participants  then  loaded  a  similar  dataset 

17 


to  the  previous  session  to  work  with  through  the  first  several  minutes  of  the  tutorial.  The 
first  new  technique  taught  was  loading  a  Shapefile  with  predefined  polygons,  centering 
the  interface  on  that  area  of  the  screen,  and  then  viewing  and  renaming  graphs  populated 
in  those  polygons.  The  subjects  were  then  instructed  on  methods  for  making  user  defined 
polygons  for  specific  smaller  regions  of  interest,  and  the  accompanying  statistical  output. 
The  next  training  item  covered  was  the  loading  and  use  of  map  layers  within  the  Map  and 
Data  Display  Area,  and  another  method  for  user  defined  polygons.  Using  the 
accompanying  graphs,  the  remaining  options  for  graph  modification  were  covered. 

5.  Session  Three 

The  third  and  final  session  included  no  training;  rather  it  was  focused  on 
conducting  a  sequence  of  tasks.  There  were  several  purposes  for  the  last  session.  The 
first  was  to  observe  the  workflow  of  the  participants  as  they  conducted  tasks.  This  was 
done  primarily  via  eye-tracking  with  the  EyeWorks  software,  and  by  the  administrators 
of  the  experiment  recording  subject’s  answers  to  the  final  two  tasks.  The  complete  list  of 
tasks  for  session  three  is  contained  in  Appendix  E.  It  began  with  tasks  one  through  eight 
as  a  series  of  step-by-step  simple  tasks  such  as  navigating  to  Africom  and  Iraq,  loading 
data  and  Shapefiles,  and  creating  user-defined  polygons.  The  intent  was  to  refresh  the 
software  and  functionality  of  the  interface.  The  ninth  task  removed  all  current  data  sets 
and  Shapefiles,  and  then  loaded  the  sets  for  the  final  two  tasks.  Task  ten  was  a  more 
complex  and  comprehensive  task  that  required  the  participant  to  work  through  several 
steps  without  having  the  step-by-step  questioning  entered  into  the  EyeWorks  software  to 
direct  them  through  completion.  The  objective  was  for  the  subject  to  complete  a  task 
requiring  a  full  series  of  actions  from  loading  a  data  set  to  generating  a  graph  that  could 
be  used  to  answer  a  theoretical  commander’s  question.  The  participant  was  directed  to 
determine  the  total  number  of  IED  attacks  for  2010  in  the  most  northwestern  provincial 
boundary  of  Afghanistan.  Essentially,  this  task  began  with  a  blank  slate  and  required 
loading  a  dataset  and  Shapefile,  then  navigating  the  Map  and  Data  Display  Area  to 
Afghanistan  by  a  means  of  their  choice,  and  generating  and  opening  a  specific  graph. 
Subjects  were  then  required  to  answer  the  question  about  that  graph.  If  a  subject  could 
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not  complete  the  tasks,  he  or  she  was  unable  to  answer  the  question.  The  final  task  was  a 
trend  analysis  question  that  also  required  the  completion  of  all  of  task  number  ten  to 
answer.  Answers  to  questions  ten  and  eleven  were  recorded  by  the  administrators  of  the 
experiment.  Following  the  completion  of  task  eleven,  all  work  within  DaViTo  was 
complete.  The  participant  then  proceeded  to  the  final  series  of  graphing  questions,  with 
answers  recorded  by  EyeWorks  Record.  The  final  portion  of  the  session  consisted  of  an 
extensive  questionnaire  of  50  questions  composed  from  the  seven  usability  factors. 

6.  Experiment  Scheduling  Considerations 

The  overall  design  of  the  experiment  was  for  it  to  take  place  over  a  three  week 
timespan  for  each  participant,  within  a  six  week  window  of  open  scheduling  available. 
This  gave  subjects  the  flexibility  to  choose  a  four  week  period  in  which  to  conduct  their 
first  session  and  still  have  two  weeks  following  that  for  the  remaining  two  sessions. 
Subjects  were  to  schedule  their  sessions  no  less  than  six  days  and  no  more  than  eight  days 
apart.  The  goal  was  seven  days  between,  but  plus  or  minus  one  day  was  allowed  for 
flexibility.  Finally,  sessions  were  available  throughout  the  Monday  through  Friday 
workweek,  from  0700  to  1800  daily. 

D.  SUBJECTS 

In  accordance  with  the  approved  IRB,  all  subjects  were  taken  from  the  pool  of 
personnel  of  the  Naval  Postgraduate  School  (NPS).  This  included  the  student  population, 
civilians,  and  foreign  officers  with  only  two  conditions.  The  first  condition  was  that  a 
participant  could  not  be  colorblind  due  to  the  coloring  of  the  charts  used  in  DaViTo.  The 
second  required  that  the  individual  not  be  involved  in  the  development  of  the  DaViTo 
software.  Participation  in  the  study  was  completely  voluntary  and  it  was  made  clear 
during  recruitment  and  throughout  the  experiment  that  the  subjects  were  free  to  stop  at 
any  time,  as  well  as  the  fact  that  there  would  be  no  compensation  for  participation.  The 
demographic  survey  for  the  subject  backgrounds  is  contained  in  Appendix  F. 

The  experiment  sessions  were  scheduled  to  be  conducted  over  a  six  week  period, 
on  weekdays  between  0700  and  1800.  When  scheduling  their  three  sessions,  subjects 
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were  directed  to  sign  up  for  one  hour  blocks  in  three  consecutive  weeks,  preferably  seven 
days  apart,  but  within  a  window  of  six  to  eight  days  between  sessions.  Almost  all 
subjects  signed  up  for  sessions  with  seven  days  of  separation  between  them,  and  all 
sessions  were  scheduled  with  the  six  to  eight  day  gap  between  sessions.  Every  subject’s 
sessions  were  completed  in  five  weeks  and  the  last  week  allotted  for  experiments  did  not 
need  to  be  utilized. 

The  experiment  was  limited  to  24  subjects  in  order  to  have  an  adequate  amount  of 
data  for  statistical  analysis,  but  to  also  have  enough  time  to  conduct  72  hours  of  sessions. 
Recruitment  was  conducted  through  mass  e-mail,  word  of  mouth,  and  via  classroom 
briefs.  The  IRB  approved  recruitment  document  is  contained  in  Appendix  F.  The  goal  of 
24  subjects  was  met  through  the  recruitment  efforts,  with  a  fairly  diverse  pool.  The 
average  participant  age,  as  shown  in  Table  3,  was  34.92,  consisted  almost  entirely  of  O- 
3s  and  0-4s,  presented  in  Table  4,  and  the  dominant  service,  seen  in  Table  5,  was  the 
U.S.  Navy.  The  demographic  survey  responses  also  permitted  the  subjects  to  assess  their 
computer  skill  level  and  average  usage.  Nearly  one-third  of  subjects  rated  their  skill  level 
at  intermediate,  as  seen  in  Table  6,  while  Table  7  shows  that  half  the  subjects  indicated 
usage  in  excess  of  2 1  hours  per  week.  Other  items  of  note  from  the  demographic  survey 
include  a  majority  of  subjects  having  a  preference  for  the  Windows  operating  system, 
shown  in  Table  8,  seven  subjects  had  some  unspecified  amount  of  previous  experience 
with  the  mapping  and  spatial  analysis  tool  ArcGIS,  and  no  subjects  had  any  knowledge  of 
the  DaViTo  software. 


Subjects  Age  (years) 

Min 

28 

Max 

42 

Average 

34.92 

Mode 

40 

Median 

35 

Standard  Deviation 

4.57 

Table  3.  Breakdown  of  subject  population  by  age. 
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Number  of  Subjects 
By  Rank 

02 

0 

03 

11 

04 

12 

05 

1 

06 

0 

Table  4.  Breakdown  of  subject  population  by  rank. 


Number  of  Subjects  By 
Service 

Army 

7 

Navy 

12 

Marine  Corps 

4 

Air  Force 

0 

Retired 

0 

International 

1 

Table  5.  Breakdown  of  subject  population  by  rank. 


Computer  Skill  Level 

Beginner 

1 

Novice 

6 

Intermediate 

15 

Expert 

2 

Table  6.  Subjects’  assessment  of  their  own  computer  skill  level. 
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Subjects  Hours  Per  Week 

Using  a  Computer 

(1)  Hours  0-5 

1 

(2)  Hours  6-10 

0 

(3)  Hours  11-15 

5 

(4)  Hours  16-20 

6 

(5)  Hours  21+ 

12 

Average  Response 

4.17 

Mode 

5 

Median 

4.5 

Standard  Deviation 

1.03 

Table  7.  Subjects  reported  weekly  computer  usage. 


Preferred  Operating  Sysf 

:em 

Mac 

4 

Windows 

19 

Other 

1 

Table  8.  Summation  of  subject  populations  preferred  operating  system. 


E.  EXPERIMENT  PROCEDURE 

The  experiment  consisted  of  three  sessions,  and  the  procedure  was  common  for 
all  three  sessions.  The  difference  between  sessions  was  content  as  covered  previously. 
For  the  subjects,  there  was  no  difference  between  experimental  setup  and  procedure 
except  for  in  sessions  one  and  two  they  controlled  the  playback  of  the  tutorial,  whereas  in 
session  three  there  was  no  tutorial. 

Prior  to  use  each  day  it  was  necessary  to  check  the  calibration  of  the  eye-tracking 
cameras  and  software.  If  required,  a  full  system  calibration  was  conducted. 
Additionally,  stereo  and  IR  camera  positions  were  verified  to  be  correct.  DaViTo  was 
then  started  and  checked  for  proper  function  in  case  of  any  system  freezes. 

Upon  arrival  for  session  one,  subjects  were  directed  to  read  the  Consent  to 
Participate  in  Research  form,  contained  in  Appendix  F,  and  to  sign  and  date  it  if  in 
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agreement.  The  form  was  then  signed  by  the  researcher  as  a  witness.  Following  that, 
they  were  directed  to  complete  a  demographic  questionnaire,  contained  in  Appendix  F. 
The  researcher  instructed  the  subject  to  sit  in  front  of  the  cameras  and  screens  in  a 
comfortable  position.  They  were  then  linked  up  with  faceLAB  and  eye-tracking 
commenced,  stepping  through  the  steps  of  the  software  to  properly  track  facial 
components  such  as  the  corners  of  the  eyes  and  mouth  in  order  to  optimize  the  tracking 
performance.  FaceLAB  tracking  was  then  checked  by  following  a  series  of  dots  on  the 
screen.  This  data  was  saved  by  taking  a  screenshot  and  recording  it  on  the  experiment 
checklist.  Once  complete,  control  was  then  shared  with  EyeWorks  and  an  additional  dot 
check  was  done  by  having  the  participant  follow  dots  on  the  screen.  This  completed 
setup  of  the  participant  with  the  eye-tracking  software.  Before  beginning  the  experiment, 
the  subject  was  given  final  instruction  on  how  to  use  the  left  hand  mouse  for  controlling 
the  tutorial  on  the  left  screen,  and  how  to  use  the  right  hand  mouse  for  DaViTo  and  the 
EyeWorks  software  that  controlled  the  experiment  when  moving  from  DaViTo  to  graph 
and  survey  questions. 

The  actual  experiment  was  now  ready  to  begin.  The  experiment  proctor  would 
push  record  on  the  EyeWorks  interface  and  instruct  the  participant  to  begin.  For  sessions 
one  and  two,  the  participants  were  directed  within  the  EyeWorks  window  to  follow  along 
with  the  tutorial,  and  it  would  then  exit  out  to  the  DaViTo  interface.  For  these  sessions, 
upon  completion  of  the  tutorial  they  would  go  back  to  the  EyeWorks  window.  The 
subject  would  then  proceed  through  the  graphing  and  survey  questions  until  complete.  At 
this  time  recording  in  EyeWorks  would  be  stopped,  the  data  saved,  and  the  subject 
reminded  of  their  future  session  times.  Similarly,  for  the  final  session  the  experiment 
would  begin  in  the  EyeWorks  window  with  the  initial  task.  The  participant  would 
proceed  through  all  of  the  tasks,  transition  to  graphing  questions,  and  conclude  with  a 
longer  series  of  survey  questions.  The  subject  was  then  thanked  for  their  participation 
and  dismissed.  The  proctor  saved  the  information  under  the  participant’s  number  and 
proceeded  to  setup  the  system  for  the  next  session. 
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F.  DATA  COLLECTION  AND  ANALYSIS  STRATEGY 

1.  Eye-Tracking  Data 

This  data  collection  was  method  was  used  during  all  three  sessions.  The  eye¬ 
tracking  data  that  was  of  the  highest  importance  was  that  of  session  three,  particularly  the 
conduct  of  completing  the  tasks.  Task  completion  for  the  first  two  sessions,  as  well  as 
graph  evaluation  by  the  subjects  was  also  of  importance. 

While  there  are  a  large  number  of  analysis  techniques  available  through 
EyeWorks  Analyze,  only  a  few  were  used  with  the  goal  of  delivering  recommendations 
to  TRAC  Monterey  for  improvements  to  DaViTo.  Raw  mouse  click  data  collected  for 
each  subject  and  session  was  used  extensively,  particularly  in  the  final  session  for 
comparison  to  an  efficiency  baseline.  Subject  time  to  complete  tasks  was  used  for 
comparison  to  a  time  baseline.  Additionally  fixation  data  was  utilized  in  the  form  of  heat 
maps  to  determine  visual  concentration  areas  of  the  DaViTo  interface. 

2.  Graphs 

Data  representation  research  was  important  to  see  if  the  graphical  methods 
employed  by  default  in  DaViTo  are  the  appropriate  way  to  display  data,  as  well  as 
evaluate  some  similar  methods  of  data  representation.  The  current  default  is  the  line 
chart  fonnat,  with  the  stacked  bar  format  also  available,  but  there  are  a  myriad  of 
possibilities  with  the  inclusion  of  JFreeChart  and  the  R  statistical  package.  These  include 
but  are  not  limited  to  the  stacked  line  with  markers  fonnat,  the  stacked  area  format,  and 
the  clustered  columns  format,  as  well  as  many  others.  To  evaluate  this,  at  the  conclusion 
of  each  series  of  tasks  the  participant  was  then  directed  within  the  EyeWorks  Record 
session  to  answer  similar  questions  about  a  number  of  graphs.  Their  answers  were  saved 
by  EyeWorks  Record. 

Participant  answers  were  later  imported  to  Excel  and  used  in  multiple  analyses 
with  Excel  and  JMP.  The  strategy  was  to  evaluate  the  resultant  data  for  efficiency,  with 
time  to  complete  and  correctness  the  parameters  of  interest. 
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3. 


Questionnaires 


The  participant  answers  to  the  surveys  were  recorded  by  EyeWorks  Record,  and 
then  imported  to  Excel  for  analysis.  The  multiple  choice  questions  were  evaluated  on  the 
basis  of  average,  mode,  median,  and  frequency.  The  free  entry  questions  were  evaluated 
for  similar  responses  and  used  to  formulate  participant  recommendations  for 
improvement. 
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IV.  RESULTS  AND  DATA  ANALYSIS 


A.  DATA  PREPARATION 

Over  4.5  gigabytes  of  data  was  collected  throughout  all  three  subject  sessions. 
One  challenge  was  detennining  how  to  partition  the  data  such  that  it  could  be  analyzed 
and  presented  in  a  logical  and  efficient  manner.  Another  was  which  data  to  use, 
particularly  the  eye-tracking  output  of  the  EyeWorks  software,  as  there  are  a  number  of 
analytic  options  and  some  data  was  not  directly  relevant  to  this  thesis  or  its  scope.  The 
purpose  was  to  provide  actionable  recommendations  to  TRAC  Monterey  to  improve  the 
usability  and  workflow  of  the  DaViTo  software.  As  such,  not  all  resultant  data  was 
analyzed  and  will  not  be  discussed  here. 

The  sessions  had  distinct  divisions  which  made  for  a  natural  separation  of  the  data 
into  three  sections.  The  purpose  of  the  first  interaction  with  the  DaViTo  software 
following  the  tutorials  and  initially  in  the  beginning  of  session  three  was  to  conduct  tasks; 
which  became  one  grouping  of  information  for  data  analysis.  Following  task  completion, 
the  subjects  would  analyze  graphs  and  answer  questions  in  order  to  determine  the  best 
data  representation  method.  This  became  an  additional  grouping  for  analysis.  Finally, 
the  experiment  subjects  completed  a  series  of  questions  at  the  end  of  each  session.  These 
questionnaire  answers  became  the  third  grouping  for  analysis. 

B.  DAVITO  TASK  PERFORMANCE  AND  INTERACTION 

The  first  two  session  tutorials  concluded  with  a  series  of  tasks  and  the  final 
session  began  with  similar  tasks.  Task  completion,  meaning  the  user  completed  the 
directed  task,  as  well  as  time  and  correctness,  were  used  to  evaluate  the  ability  of  the 
participant  to  utilize  the  basic  skills  learned  from  the  tutorials,  such  as  navigation,  file 
operations,  and  creating  a  data  representation.  This  was  not  an  effort  to  evaluate 
participants  from  a  training  transfer  perspective,  but  rather  to  determine  the  efficiency 
and  workflow  with  which  simple  tasks  could  be  completed  by  a  novice  user  with  minimal 
training  on  the  basic  functionality  and  tools  available  in  DaViTo.  Tasks  in  the  first  two 
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sessions  were  used  to  reinforce  the  skills  taught  by  the  tutorial,  and  will  not  be  discussed. 
The  final  session  required  execution  of  a  series  of  eight  tasks  with  individual  instructions, 
followed  by  a  more  complex  task  divided  into  three  general  instructions.  Of  the  24 
participants,  only  one  achieved  a  perfect  score  by  completing  tasks  ten  and  eleven  with 
correct  answers.  Previous  incorrect  answers  in  tasks  one  through  eight  would  not  affect 
completion  of  tasks  nine  through  eleven,  however  failure  to  properly  open  the  data  set 
and  Shapefile  in  task  nine  would  prohibit  completion  of  task  ten.  No  partial  credit  was 
given. 

Three  separate  analyses  were  conducted  on  the  session  three  task  data:  number  of 
mouse  clicks  for  each  task,  duration  of  time  to  complete  a  task,  and  cumulative  heat 
mapping  of  the  screen  area  for  each  task.  Mouse  click  data  and  time  duration  of  task 
completion  were  analyzed  to  determine  efficiency  of  operation  with  the  interface. 
Cumulative  gaze  density  mapping  was  analyzed  for  workflow  processes,  showing  gaze 
concentration  areas  in  a  heat  map  to  determine  if  subjects  were  looking  at  the  appropriate 
part  of  the  screen  for  completing  a  task. 

1.  Analysis  of  Mouse  Clicks,  Correctness,  and  Completeness  of  Final 
Session  Tasks 

Mouse  clicks  for  each  participant  were  recorded  automatically  by  the  EyeWorks 
software.  This  was  compared  to  a  benchmark  number  of  mouse  clicks  established  by  the 
experiment  designers  as  the  most  efficient  possible  manner  to  complete  the  tasks 
correctly.  Task  descriptions  for  session  three  are  in  Appendix  E. 

The  summary  results  for  the  tasks  in  session  three  are  displayed  in  Table  9.  This 
is  also  graphically  represented  in  Figures  3  and  4.  The  most  notable  result  from  the 
mouse  click  analysis  was  the  large  deviation  from  the  baseline  for  tasks  four  and  ten. 
Task  four  directed  the  participants  to  “create  a  rectangle  chart  on  the  data  point  in  the 
southwestern  most  part  of  Kenya  and  display  the  chart”.  As  seen  in  Table  9  the 
benchmark  for  this  task  was  10  mouse  clicks,  yet  the  average  number  of  clicks  for 
subjects  was  25.43,  with  a  standard  deviation  of  12.47.  This  difference  from  benchmark 
to  subject  average  is  displayed  in  Figure  3.  While  users  who  completed  the  task  correctly 
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improved  to  22.63  clicks,  the  average  was  far  worse  for  users  that  could  not  complete  the 
task  correctly,  with  31.86  clicks,  as  shown  in  Figure  4.  Task  ten  was  a  more  complex 
task,  directing  the  subjects  to  “please  answer  the  following  question:  what  were  the  total 
events  for  IED  attacks  for  2010  in  the  most  northwestern  provincial  boundary”.  Table  9 
shows  that  on  average  subjects  clicked  15.87  times  to  complete  the  task,  while  the 
benchmark  was  5  clicks,  with  a  standard  deviation  of  12.68.  This  deviation  between  the 
benchmark  and  subject  average  is  shown  in  Figure  3.  Figure  4  shows  that  for  subjects 
who  did  not  complete  the  task  correctly  they  averaged  only  15  clicks,  while  those  who 
did  averaged  35  clicks.  This  indicates  that  some  subjects  gave  up  on  task  ten.  Both  tasks 
required  the  subjects  to  make  a  shape  within  which  a  graph  would  be  made.  Once  the 
shape  is  made  the  user  must  click  outside  of  the  shape  to  get  the  graph  to  populate  within 
it.  Frequently  when  doing  this  the  program  would  lag,  and  without  a  progress  status, 
users  would  be  repeatedly  clicking  outside  the  shape  in  an  effort  to  get  the  graph  to 
populate.  Some  participants  even  gave  up  waiting  and  moved  on  to  the  next  task, 
particularly  for  task  ten.  This  is  indicative  of  a  usability  deficiency  within  the  software. 


Task 

Number 

Bench 

Mark 

Average 
Mouse  Clicks 
for  All  Users 

Average  Mouse 
Clicks  for 
Correct  Answers 

Average 
Mouse  Clicks 
for  Incorrect 
Answers 

Min 

Max 

Std 

Dev 

Task  1 

4 

6.52 

6.52 

0 

5 

19 

3.31 

Task  2 

3 

7.00 

7.00 

0 

4 

36 

8.62 

Task  3 

3 

3.48 

3.48 

0 

1 

8 

1.27 

Task  4 

10 

25.43 

22.63 

31.86 

7 

53 

12.47 

Task  5 

3 

4.13 

4.33 

2 

1 

15 

2.96 

Task  6 

3 

5.30 

5.32 

5 

3 

22 

4.17 

Task  7 

3 

6.04 

6.04 

0 

3 

31 

6.59 

Task  8 

4 

5.70 

5.50 

5.91 

3 

15 

3.43 

Task  9 

14 

19.09 

18.35 

24 

12 

42 

7.88 

Task  10 

5 

15.87 

35.00 

15 

1 

45 

12.68 

Task  11 

2 

3.13 

2.42 

6.5 

1 

23 

4.68 

Table  9.  Summary  of  mouse  click  and  correctness  statistics  for  all  session  three 
participants. 
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Figure  3.  Graph  comparing  average  participant  mouse  clicks  to  the  bench  mark. 


Mouse  Clicks  for  Complete  and  Incomplete  Tasks 


♦  Average  Mouse  Click 
For  Completed  Task 

■  Average  Mouse  Clicks 
For  Incomp  ete  lask 

A  Bench  Mark 


Figure  4.  Graph  comparing  the  bench  mark  to  average  participant  mouse  clicks  for 
both  complete  and  incomplete  session  three  tasks. 


2.  Analysis  of  Time  Duration,  Correctness,  and  Completeness  of  Final 
Session  Tasks 

The  time  for  every  participant  to  complete  each  session  three  task  was 
automatically  recorded  by  the  EyeWorks  software.  This  was  then  compared  to  a 
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benchmark  time  established  by  the  experiment  designers  as  the  most  efficient  possible 
time  in  which  to  complete  the  tasks  correctly.  Task  descriptions  for  session  three  are  in 
Appendix  E. 

The  summary  results  for  the  duration  of  tasks  in  session  three,  and  percent  correct 
are  displayed  in  Table  10.  The  most  notable  result  from  the  analysis  of  time  to  complete 
tasks,  and  similar  to  the  mouse  click  analysis,  was  the  large  deviation  from  the  baseline 
for  tasks  four  and  ten,  shown  in  Figure  5.  Again,  task  four  directed  the  participants  to 
“create  a  rectangle  chart  on  the  data  point  in  the  southwestern  most  part  of  Kenya  and 
display  the  chart.”  While  the  bench  mark  for  task  four  was  62.09  seconds,  on  average  it 
took  subjects  more  than  100  seconds  longer  to  complete  it,  with  a  standard  deviation  of 
67.02,  shown  in  Table  10.  There  was  little  deviation  between  users  who  were  able  to 
correctly  complete  task  four  from  those  who  could  not,  but  the  large  deviation  for  both  is 
shown  in  Figure  6.  Task  ten  was  a  more  complex  task,  directing  the  subjects  to  “please 
answer  the  following  question:  what  were  the  total  events  for  IED  attacks  for  2010  in  the 
most  Northwestern  provincial  boundary.”  Similar  to  the  mouse  click  data,  Table  10 
shows  a  large  deviation  from  the  baseline  of  63.36  seconds  for  this  task.  The  average 
user  completed  this  task  in  141.47  seconds  with  a  standard  deviation  of  76.17.  This  time 
was  even  longer  for  subjects  who  completed  the  task  correctly,  with  Figure  6  showing  the 
deviation  between  the  baseline,  incorrect,  and  correct  task  completion.  Note  that  task 
completion  refers  to  finishing  the  task,  not  answering  the  question  correctly.  Only  one 
subject  was  able  to  answer  the  question  asked  in  task  ten  correctly.  Both  tasks  required 
the  subjects  to  make  a  shape  within  which  a  graph  would  be  made.  Once  made,  the  user 
must  click  outside  of  the  shape  to  get  the  graph  to  populate  within  it.  DaViTo  would 
frequently  lag  when  doing  this,  and  without  a  progress  status,  users  would  repeatedly 
click  outside  the  shape  in  an  effort  to  get  the  graph  to  populate.  Confusion  from  the  delay 
and  no  status  indication  caused  some  subjects  to  give  up,  having  no  idea  if  the  software 
was  processing  their  command,  and  is  indicative  of  a  usability  deficiency. 
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Session  3 
Task 

Bench 

Mark 

Average  Time  Per 
Task  (sec) 

Min 

Max 

Std  Dev 

Percent 

Correct 

Task  1 

10.72 

25.38 

9.50 

78.13 

17.17 

100 

Task  2 

42.96 

52.66 

10.53 

202.92 

36.46 

100 

Task  3 

9.55 

14.86 

3.25 

48.66 

9.27 

100 

Task  4 

62.09 

165.69 

52.28 

319.81 

67.02 

70 

Task  5 

8.87 

17.34 

2.30 

66.99 

15.12 

91 

Task  6 

6.77 

31.18 

8.68 

96.76 

21.27 

96 

Task  7 

8.62 

28.34 

9.02 

140.33 

34.70 

100 

Task  8 

11.75 

33.77 

9.23 

149.94 

35.50 

52 

Task  9 

90.93 

90.61 

24.18 

245.10 

46.46 

87 

Task  10 

63.36 

141.47 

7.17 

355.25 

76.17 

4 

Task  11 

16.63 

16.72 

4.45 

281.58 

57.54 

83 

Table  10.  Summary  of  time  duration  statistics  for  all  session  three  participants. 
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Figure  5.  Graph  comparing  average  participant  task  durations  to  the  bench  mark. 
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Task  Duration  for  Complete  and  IncompleteTasks 


♦  Bench  Mark 
■  Completed  Tasks 
▲  Incompleted  Task 


Figure  6.  Graph  comparing  the  bench  mark  to  average  participant  time  duration  for 
both  complete  and  incomplete  session  three  tasks. 


3.  Analysis  of  Cumulative  Gaze  Frequencies  of  Final  Session  Tasks 

Along  with  time  duration  and  number  of  mouse  click  to  complete  each  task,  the 
EyeWorks  Analyze  software  suite  has  the  capability  to  generate  heat  maps  and  associated 
graphs.  The  heat  maps  were  useful  to  display  the  gaze  concentration  of  each  participant, 
or  cumulatively  for  each  task.  The  gaze  observation  bar  graphs  were  then  used  to 
compare  gaze  concentration  in  the  designated  regions  of  the  user  interface,  similar  to 
those  represented  in  Figure  2.  Task  descriptions  for  session  three  are  in  Appendix  E. 
Heat  maps  and  gaze  concentration  graphs  for  all  eleven  tasks  conducted  in  session  three 
are  in  Appendix  B.  Note  that  on  the  heat  map,  color  increasing  from  blue  to  yellow  and 
then  to  red  indicates  a  greater  number  of  fixations.  Also,  there  are  two  numbers 
displayed  in  each  region  on  the  heat  map.  The  top  number  correlates  to  the  percentage  of 
total  time  the  participants  looked  in  that  region.  The  bottom  number  is  the  percentage  of 
total  clicks  that  were  made  by  subjects  in  that  region. 
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Following  analysis  of  the  heat  maps  and  gaze  observation  graphs  for  each  task, 
three  of  the  maps  were  found  to  be  of  specific  interest.  In  addition  to  reinforcing  the 
conclusions  reached  from  the  mouse  click  and  time  duration  data  with  regards  to  tasks 
four  and  ten,  an  additional  deficiency  was  found  in  task  one. 

Task  one  was  a  simple  task  that  had  been  completed  in  sessions  one  and  two  with 
the  tutorial.  The  user  was  instructed  to  open  the  file  WITS.csv  in  order  to  populate  the 
Map  and  Data  Display  Area  with  a  dataset.  Figures  7  and  8  show  the  gaze  data  for  task 
one.  This  required  the  user  to  open  the  file  using  the  Data  Set  region  of  the  interface. 
Note  in  Figure  8  that  while  this  was  the  predominant  region  where  gazes  were  observed, 
a  significant  number  were  also  recorded  in  the  Map  area  as  well  as  the  Map  Controls 
area.  Figure  7  also  displays  a  large  concentration  of  users  gazing  at  the  Map  Controls 
area,  and  the  mouse  icons  representing  mouse  clicks  there.  It  is  expected  that  there  are  a 
large  number  of  gazes  in  the  Map  area  due  to  its  size  and  the  fact  that  the  user  would 
focus  or  scan  there  while  waiting  for  the  dataset  to  populate.  However,  the  mouse  clicks 
and  concentration  of  gaze  in  the  Map  Controls  area  indicates  that  participants  showed  a 
tendency  to  go  to  this  region  of  the  interface  when  trying  to  open  a  file.  This  is  consistent 
with  other  software  programs  that  have  a  menu  ribbon  at  the  top  of  the  page  with 
functionality  to  open  a  file,  but  there  is  no  functionality  such  as  this  in  DaViTo.  This 
again  indicates  a  usability  deficiency  of  the  DaViTo  software. 

Finally,  the  conclusions  reached  from  the  task  duration  and  mouse  clicks  data  is 
further  reinforced  by  the  heat  maps  for  task  four,  shown  in  Figure  9,  and  task  ten,  shown 
in  Figure  10.  Tasks  four  and  ten  required  subjects  to  create  a  user  defined  polygon,  and 
allow  it  to  populate  with  a  graph  of  activity  within  the  polygon.  These  heat  maps  show 
the  large  number  of  mouse  clicks  during  these  tasks,  as  indicated  by  the  mouse  icons  in 
the  heat  maps.  This  illustrates  the  unnecessary  mouse  clicking  that  was  done  during 
these  tasks.  Additionally,  these  tasks  have  significant  gaze  concentrations  in  the  Map 
Controls  area,  indicating  that  users  may  be  searching  there  for  the  shape  drawing 
functionality  that  requires  right  clicking  in  the  Map  and  Data  Display  area.  This  further 
supports  the  usability  deficiencies  previously  discussed  in  this  chapter  with  regard  to 
excessive  mouse  clicking  and  missing  functionality  in  the  Map  Controls  ribbon. 
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Figure  7.  Final  session  task  one  heat  map  and  mouse  clicks.  Task  one  directed  subjects  to  open  the  WITS.csv  data  file,  which 
required  interaction  with  the  Data  Sets  region. 
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Gaze  Observations  in  Each  Region 


.. _ 1 1  -1 _ 

Regions 


mm  dmsw(1) 
mm  MMTtfMTMft  (1) 

■  MkID 
■■  VicCorooum 
Itogra**  toll) 

■■  K*ffoc  S«»*c»o»>  01 

fUr  («vrd»  1 1) 


Figure  8.  Graph  of  final  session,  task  one,  cumulative  counts  of  gaze  observations  for 
all  subjects.  Counts  correlate  to  the  number  of  subject  fixations  in  each 
region  shown  in  Figure  7. 
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Figure  9.  Final  session  task  four  heat  map  and  mouse  clicks.  Task  four  directed  subjects  to  create  a  user  defined  polygon  and 
display  the  chart. 
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Figure  10.  Final  session  task  ten  heat  map  and  mouse  clicks.  Task  ten  directed  subjects  to  answer  a  question  that  required  the 
creation  of  a  user  defined  polygon,  and  to  display  the  chart. 
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4. 


Task  Results  Summary 


Two  usability  deficiencies  were  identified  by  the  output  data  from  the  session 
three  tasks.  The  first  was  identified  by  the  excessive  mouse  clicks  when  attempting  to 
generate  a  graph  within  a  user-defined  polygon.  This  deficiency  is  related  to  the  attribute 
of  efficiency.  The  quickness  with  which  a  user  can  complete  their  goal  is  reduced  when 
time  is  wasted  awaiting  program  response.  Since  the  product  does  not  react  in  the 
expected  manner,  effectiveness  is  also  reduced.  The  survey  results  will  also  show  that 
these  issues  decreased  the  attribute  of  user  satisfaction.  The  second  deficiency  was 
identified  by  gaze  concentrations  indicated  on  heat  maps  in  the  improper  areas.  This 
indicates  a  lack  of  functionality  that  the  user  expects  in  the  header  menu  bar  ribbon,  and 
is  indicative  of  reduced  effectiveness  of  the  DaViTo  software.  Recommendations  for 
improvement  are  discussed  in  Chapter  V,  and  a  consolidated  list  in  Appendix  A. 

C.  GRAPH  PERFORMANCE  RESULTS 

A  series  of  data  representation  questions  followed  the  task  completion  sections  for 
all  three  sessions.  This  portion  of  the  experiment  was  used  to  evaluate  the  ability  of  the 
participant  to  interpret  graphical  output  similar  to  that  already  used  in  DaViTo,  or  able  to 
be  used.  The  purpose  was  to  ensure  that  data  is  being  represented  in  the  best  manner  for 
the  user  to  consistently  interpret  it  quickly  and  accurately.  It  should  be  noted  that  after 
session  one,  a  brief  training  session  was  given  by  the  experiment  administrators,  followed 
by  a  review  of  their  performance.  The  purpose  was  to  see  how  a  user  would  interpret  the 
graphs  with  no  training  on  their  fonnat.  Again,  this  was  not  an  effort  to  evaluate 
participants  from  a  training  transfer  perspective,  but  rather  an  attempt  to  detennine  which 
graphs  best  represented  the  type  of  data  being  used  in  DaViTo.  It  is  recognized  that  a 
great  deal  of  research  has  been  done  on  data  representation  methods  and  there  are  novel 
concepts  in  literature  that  could  be  applied  to  future  applications;  however,  this  was  an 
effort  to  compare  the  currently  used  methods  with  some  others  that  are  easily  achievable 
with  the  current  software.  Complete  statistical  analysis  of  all  graph  data  is  contained  in 
Appendix  C. 
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JFreeChart  and  R  have  the  capability  to  present  data  in  a  number  of  different 
graphical  formats.  These  different  types  are  accessible  within  DaViTo  by  an  advanced 
user;  however,  the  novice  user  has  two  readily  accessible  graph  style  options.  The 
default  option  is  the  line  chart  format  (LCF),  shown  in  Figure  11.  The  LCF  has  options 
for  line  chart  shapes  or  markers  on  (default)  or  off,  as  well  as  the  ability  to  manipulate  the 
line  thickness.  The  other  option  is  to  use  the  cumulative  stacked  bar  format  (SBF)  shown 
in  Figure  12.  These  options  are  selectable  in  the  display  region,  shown  in  Figure  2,  of  the 
DaViTo  interface.  Additionally,  data  can  be  exported  from  DaViTo  for  analysis  with 
other  statistical  tools  such  as  Excel  or  JMP. 
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Figure  1 1 .  Example  line  chart  format  (LCF)  with  markers  from  the  DaViTo  software. 
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Figure  12.  Example  cumulative  stacked  bar  format  (SBF)  from  the  DaViTo  software. 

Five  different  graph  types  were  constructed  for  the  data  representation  portion  of 
the  experiment.  Each  format  is  summarized  in  Table  11.  In  the  experiment  sessions 
there  were  questions  about  each  graph  format,  for  which  the  data  was  randomly  generated 
from  session  to  session.  These  questions  are  listed  in  Table  12  with  a  letter  designator. 
This  allowed  the  use  of  the  same  questions  for  each  graph,  but  each  had  a  different 
answer.  These  graph  types  were  chosen  because  two  of  them  are  already  available  for 
selection  by  the  novice  user,  and  DaViTo  has  the  capability  to  easily  generate  the  others. 
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Type  and 
Abbreviation 


Description 


Example 


Line  Chart 
Format 
(LCF) 


Standard  line  chart, 
no  markers  for 
inflection  points  or 
delineators. 


Stacked  Line 
Marker 
Format 
(SLMF) 


Line  chart  with 
markers  for 
delineators,  and 
cumulative  in  the  y 
direction. 


— —  Event  D 
— a—  Event  C 
— *-  Event  B 


Event  A 


Stacked 
Area  Format 
(SAF) 


Filled  area  chart, 
no  markers  for 
inflection  points  or 
delineators,  and 
cumulative  in  the  y 
direction. 


m  cn  m 


m  m  m  m  m 


m 

fi 

o 
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Type  and 
Abbreviation 


Description 


Example 


Stacked  Bar 

Cumulative  stacked 

Format 

bar  chart. 

(SBF) 


■  Event  D 

■  Event  C 

■  Event  B 

■  Event  A 


Clustered 

Columns 

Format 

(CCF) 


Columns  for  each 
data  point  are 
clustered  side  by 
side. 


■  Event  A 

■  Event  E 

■  Event  C 

■  Event  C> 


Table  1 1 .  Table  showing  the  following  information  for  the  five  graph  types  constructed 
for  the  experiment:  Type  and  abbreviation,  description,  and  an  example  of 
each. 


Question 

Designation 

Session 

Question  Text 

A 

One 

What  were  the  total  events  for  December  2012? 

B 

One 

What  event  had  the  highest  frequency  in  Jun  2013? 

C 

One 

What  month  had  the  most  events? 

D 

Two 

What  month  had  the  least  events? 

E 

Two 

Which  month  had  the  highest  frequency  for  event  D? 

F 

Three 

Compare  the  months  of  December  2012  and  December  2013. 
What  was  the  increase  or  decrease  in  total  events? 

G 

Three 

What  event  had  the  highest  sum  for  the  months  of  February 
2012  and  February  2013? 

Table  12.  List  of  questions  for  each  session  and  their  designation  letter. 
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1.  Analysis  of  Session  One  Graph  Data 


The  participants  were  introduced  to  the  five  graph  types  identified  in  Table  1 1  in 
session  one.  Three  identical  questions  were  asked  about  each  graph  type,  listed  in  Table 
12.  These  questions  were  similar  to  those  a  possible  user  in  the  field  or  analyst  would 
ask,  such  as  how  many  events  occurred  in  this  region  in  a  certain  month  this  year,  when 
attempting  to  conduct  trend  analysis  of  a  large  array  of  data.  Subject  answers  were 
evaluated  for  time  to  answer  and  accuracy  of  their  answer.  Several  subjects  exhibited 
frustration  with  some  of  the  graph  types  due  to  the  amount  of  data  contained  or  the 
format  in  which  it  was  displayed.  Upon  completion  of  session  one  each  graph  type  was 
explained  to  the  subjects,  and  their  perfonnance  reviewed.  This  was  to  determine  if 
another  graph  type  would  perform  better  with  a  simple  explanation,  such  as  one  that 
could  be  stated  as  a  mouse-over  instruction  programmed  into  DaViTo. 

The  stacked  bar  format  (SBF)  perfonned  well  in  session  one,  with  80.56  percent 
correct  answers  and  an  average  time  to  answer  of  27.8  seconds,  as  shown  in  Table  13. 
The  SBF  outperfonned  all  other  chart  types  by  more  than  25  percent  in  percent  correct, 
and  was  second  in  time  to  answer  to  the  stacked  area  format  (SAF)  by  less  than  three 
seconds.  Despite  the  slightly  slower  time  to  answer,  the  much  larger  difference  in  correct 
answers  makes  this  the  better  performing  format.  Statistics  for  all  five  chart  types  are 
shown  in  Table  13.  Figure  13  shows  a  comparison  of  the  data  with  top  performance  in 
the  upper  right  corner.  This  figure  highlights  the  large  difference  in  perfonnance  from 
the  other  fonnats,  and  the  small  time  advantage  of  SAF.  Additionally,  subjects  answered 
the  questions  quickly  with  SBF,  which  posted  the  second  fastest  average  time  to  answer  a 
question.  It  is  also  of  note  that  the  SBF  outperformed  the  other  types  in  every  question  in 
session  one,  with  these  additional  statistics  in  Appendix  C. 

The  distant  second  place  format  in  accuracy  was  the  clustered  columns  format 
(CCF),  although  it  took  the  longest  for  users  to  evaluate.  While  it  had  much  worse 
performance  on  average,  it  should  be  noted  that  on  question  B  in  Table  12,  its 
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performance  was  similar  to  that  of  the  SBF,  with  additional  statistics  in  Appendix  C.  The 
CCF  and  SAF  graphs  had  similar  accuracy  performance  to  the  LCF.  The  worst 
performance  in  accuracy  was  the  SLMF. 


Session  1 

Accuracy  (%  Correct) 

Time  (sec) 

CCF 

54.17 

58.33 

SBF 

80.56 

27.81 

LCF 

47.22 

38.38 

SLMF 

22.22 

33.52 

SAF 

51.39 

24.86 

Table  13.  Overall  statistics  for  all  graph  types  in  session  one. 
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Figure  13.  Plot  of  session  one  time  to  answer  versus  accuracy  of  answer.  Note  the 
decreasing  time  on  the  Y-axis.  Top  performance  is  in  upper  right  comer. 

2.  Analysis  of  Session  Two  Graph  Data 

In  session  two  the  participants  were  asked  the  same  series  of  questions,  listed  in 
Table  12,  about  each  of  the  five  graph  types  identified  in  Table  11.  Two  identical 
questions  were  asked  about  each  graph  type,  similar  in  type  to  those  of  session  one. 
Subject  answers  were  evaluated  for  time  to  answer  and  accuracy  of  their  answer.  Again, 
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several  subjects  exhibited  frustration  with  some  of  the  graph  types  due  to  the  amount  of 
data  contained  or  the  format  in  which  it  was  displayed.  Upon  completion  of  session  two 
there  was  no  review  of  performance  or  explanation  of  graph  formatting. 

In  session  two  all  chart  types  showed  improved  accuracy  performance  except  the 
SBF,  which  exhibited  a  significant  decrease,  as  seen  in  Table  14,  although  it  had  the 
shortest  time  to  answer.  Figure  14  shows  that  all  types  except  the  LCF  were  clustered 
near  the  upper  right  comer,  with  the  CCF  resulting  in  the  highest  number  of  accurate 
answers  in  commensurate  time  with  the  less  accurate  SBF. 

The  format  with  the  second  best  accuracy  was  the  SAF  with  75  percent  accuracy, 
and  approximately  five  seconds  more  required  to  answer  than  the  CCF.  While  on 
average  it  had  worse  performance  than  the  CCF,  it  should  be  noted  that  on  question  D, 
from  Table  12,  it  had  the  highest  performance  and  lowest  time  to  answer.  Converse  to 
this,  on  question  E  in  Table  12  it  had  the  lowest  performance,  with  the  additional 
statistics  in  Appendix  C. 


Session  2 

Accuracy  (%  Correct) 

Time  (sec) 

CCF 

83.33 

30.38 

SBF 

64.58 

28.88 

LCF 

50.00 

42.79 

SLMF 

72.92 

29.31 

SAF 

75.00 

34.62 

Table  14.  Overall  statistics  for  all  graph  types  in  session  two. 
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Session  2  Time  vs  Accuracy 


Accuracy  (%  correct) 
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Figure  14.  Plot  of  session  two  times  to  answer  versus  accuracy  of  answer.  Note  the 

decreasing  time  on  the  Y-axis.  Top  performance  is  in  the  upper  right  corner. 


3.  Analysis  of  Session  Three  Graph  Data 

In  session  three,  the  participants  were  again  asked  a  series  of  questions  from 
Table  12  about  each  of  the  five  graph  types  identified  in  Table  11.  Two  identical 
questions  were  asked  about  each  graph  type,  similar  in  type  to  those  of  sessions  one  and 
two.  Subject  answers  were  evaluated  for  time  to  answer  and  accuracy  of  their  answer. 
Once  again,  several  subjects  exhibited  frustration  with  some  of  the  graph  types  due  to  the 
amount  of  data  contained  or  the  format  in  which  it  was  displayed.  Upon  completion  of 
session  three  there  was  no  review  of  performance  or  explanation  of  graph  formatting. 

In  this  session  the  SBF  and  CCF  were  the  top  performers  in  accuracy  at  75 
percent,  although  the  CCF  took  on  average  greater  than  15  seconds  to  answer,  as  shown 
in  Table  15.  Figure  15  shows  that  SBF,  CCF,  and  SAF  were  clustered  near  the  upper 
right  comer  with  the  best  results. 
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Session  3 

Accuracy  (%  Correct) 

Time  (sec) 

CCF 

75.00 

54.62 

SBF 

75.00 

39.09 

LCF 

50.00 

87.31 

SLMF 

47.92 

60.16 

SAF 

60.42 

45.69 

Table  15.  Overall  statistics  for  all  graph  types  in  session  three. 
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Figure  15.  Plot  of  session  three  times  to  answer  versus  accuracy  of  answer.  Note  the 
decreasing  time  on  the  Y-axis.  Top  performance  is  in  upper  right  comer. 


4.  Analysis  of  Overall  Graph  Data 

The  results  in  Table  16  clearly  indicate  that  the  SBF  was  the  top  performer  in  both 
accuracy  and  time,  at  73.38  percent  and  35.06  seconds,  respectively.  The  SLMF  had  the 
worst  accuracy  and  the  LCF  the  worst  time,  also  in  Table  16.  SBF  and  LCF  formats  are 
already  in  use  in  DaViTo,  with  the  LCF  as  the  default  type.  Figure  16  shows  that  CCF 
was  a  close  second  in  accuracy,  but  on  average  takes  50  percent  more  time  to  answer  the 
question  than  with  the  SBF. 
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While  the  SBF  was  clearly  the  top  performer,  there  are  other  things  to  note  when 
looking  at  the  overall  performance.  If  only  one  type  of  graph  were  to  be  used  in  DaViTo, 
the  SBF  would  be  the  recommended  one;  however,  other  formats  have  strong  points  as 
well  and  should  be  utilized.  Figure  17  shows  the  performance  of  all  formats  for  each 
question.  Six  of  the  seven  questions  that  used  the  SBF  graph  are  clustered  in  the  upper 
right  hand  corner  showing  consistent  top  performance.  The  inconsistent  outlier  for  the 
SBF  was  question  E  at  42  percent,  which  was  one  of  the  higher  performing  questions  for 
the  LCF  at  75  percent  accuracy.  The  CCF  also  perfonned  well  on  question  E  at  87.5 
percent.  This  indicates  that  no  one  graph  type  is  appropriate  for  all  data  representations 
or  questions. 


All  Sessions 

Accuracy  (%  correct) 

Time  (sec) 

CCF 

70.83 

47.78 

SBF 

73.38 

31.93 

LCF 

49.07 

56.16 

SLMF 

47.69 

40.99 

SAF 

62.27 

35.06 

Table  16.  Overall  statistics  for  all  graph  types  in  all  sessions. 
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Figure  16.  Plot  of  all  three  sessions  times  to  answer  versus  accuracy  of  answer.  Note  the 
decreasing  time  on  the  Y-axis.  Top  performance  is  in  upper  right  comer. 
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All  Questions 
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Figure  17.  Plot  of  all  questions  times  to  answer  versus  accuracy  of  answer.  The  labels 
are  the  graph  type  followed  by  the  question  identifier.  Note  the  decreasing 
time  on  the  Y-axis.  Top  performance  is  in  upper  right  comer 

5.  Graph  Results  Summary 

Two  usability  deficiencies  were  noted  as  a  result  of  the  data  representation  portion 
of  the  experiment.  Recommendations  are  consolidated  in  Chapter  V. 

The  first  identified  deficiency  for  data  representation  is  related  to  the  attribute  of 
usefulness,  and  was  the  inability  to  answer  all  questions  with  considerably  high 
performance  with  the  default  graph  format  used  by  DaViTo.  The  default  LCF  graph  is 
the  lowest  performing  graph  of  the  five  tested.  Fortunately,  the  highest  performing 
graph,  the  SBF,  is  also  readily  available  for  use  with  the  ease  of  a  simple  radio  button  in 
the  Display  region  of  the  DaViTo  interface.  Using  the  results  of  both  of  these  graph 
formats  all  questions  were  able  to  be  answered  with  an  accuracy  that  was  on  average  in 
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excess  of  70  percent.  However,  this  puts  the  user  in  the  position  of  recognizing  that  a 
different  graph  would  relay  information  better,  and  have  the  technical  prowess  to  change 
the  graph  type.  This  also  results  in  deficiencies  in  the  attributes  of  efficiency, 
effectiveness,  and  likely  satisfaction. 

The  second  identified  deficiency  is  in  the  attributes  of  efficiency  and 
effectiveness.  As  seen  in  Figure  17,  different  graphs  perform  better  for  different  types  of 
questions,  as  well  as  different  users.  While  the  SBF  chart  performed  well  on  six  out  of 
seven  questions,  there  were  other  formats  that  performed  better  on  certain  questions.  For 
example,  the  SBF  performed  well  at  75  percent  accuracy  on  question  G;  however  it  was 
outperformed  by  CCF,  which  had  100  percent  accuracy.  In  question  by  question 
comparison,  one  can  see  which  types  perfonn  better  and  that  a  cumulative  type  graph 
such  as  SBF  is  not  always  better  for  questions  that  involve  single  event  types  rather  than 
totals,  as  no  mental  math  is  required  to  derive  the  number  from  the  stack  of  bars.  The 
results  of  individual  questions  can  be  seen  in  Appendix  C. 

The  worst  performing  graph  types  were  the  LCF  and  SLMF.  The  LCF  is  the 
default  choice  in  DaViTo,  and  can  also  be  used  with  markers  like  the  SLMF.  While  they 
had  overall  accuracy  averages  of  less  than  50  percent  and  some  of  the  longest  average 
times,  they  performed  well  at  some  questions.  This  is  again  evidence  that  having 
multiple  options  gives  users  the  capability  to  choose  the  type  that  best  conveys  the  data 
they  are  trying  to  represent. 

D.  SURVEY  RESULTS 

Each  session  was  concluded  with  a  survey  to  allow  the  participant  to  give 
feedback  about  the  training  and  software.  For  sessions  one  and  two,  there  were  three 
common  questions,  which  were  also  the  first  three  questions  for  the  final  session.  The 
final  session  concluded  with  a  more  extensive  survey  with  questions  related  to  the  five 
usability  attributes  listed  in  Table  1,  as  well  as  categories  of  improvements  and 
visualizations.  All  questions  are  contained  in  Appendix  F.  Unless  otherwise  noted, 
questions  that  were  not  free-form  text  entry  by  the  subject  used  the  number  key  shown  in 
Table  17. 
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Strongly 

Disagree 

Disagree 

Neither  Agree 

nor  Disagree 

Agree 

Strongly  Agree 

1 

2 

3 

4 

5 

Table  17.  Key  used  by  subjects  to  answer  survey  questions  that  were  not  free-form  text. 


1.  Analysis  of  the  Three  Common  Questions  from  Each  Session 

Three  standard  questions  were  used  for  the  first  two  sessions  and  were  contained 
in  the  final  session  questionnaire.  These  questions  were  standardized  across  each  session 
to  allow  for  comparison.  The  only  exception  is  question  one  for  session  three,  as  it 
included  no  training. 

The  first  common  question  asked  the  subjects  to  evaluate  the  training  during  that 
session.  The  session  three  question  was  not  evaluated  against  sessions  one  and  two 
because  there  was  no  training,  and  will  not  be  discussed.  The  training  was  evaluated  on  a 
five  point  scale  from  very  good  to  very  poor,  similar  to  Table  17.  The  results  of  common 
question  one  for  sessions  one  and  two  are  shown  in  Table  18.  This  high  valuation  may 
give  TRAC  Monterey  guidance  on  the  development  of  future  training  tools  to  distribute 
with  the  software.  Full  analysis  can  be  found  in  Appendix  D. 


Session 

Average  Response  for  Question  1:  How 
would  you  rate  the  training  session? 

1 

4.6 

2 

4.5 

Table  18.  Common  question  one  results. 


The  second  common  question  required  the  subjects  to  self-evaluate  the  amount  of 
additional  training  they  may  need  in  addition  to  that  already  received  to  feel  comfortable 
using  DaViTo  in  the  field.  This  question  was  evaluated  on  the  scale  shown  in  Table  19. 
Figure  18  shows  that  subject  confidence  decreased  between  sessions  one  to  two  as 

subjects  felt  they  needed  14  percent  more  training.  Following  the  final  session, 
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participant’s  confidence  increased  and  the  amount  of  additional  training  thought  to  be 
needed  decreased  by  21  percent  to  its  lowest  level.  This  translates  to  a  total  of  five  to 
seven  hours  of  training  are  thought  to  be  required  in  addition  to  the  two  20  minute  tutorial 
sessions.  This  response  may  give  TRAC  Monterey  guidance  on  the  development  of 
future  training  tools  and  the  amount  of  additional  training  needed.  For  full  analysis 
results  see  Appendix  D. 


0-1  Hours 

2-4  Hours 

5-7  Hours 

8-10  Hours 

10  +  Hours 

1 

2 

3 

4 

5 

Table  19.  Evaluation  scale  for  common  question  two.  This  question  asked  subjects: 

“how  many  more  hours  of  training  do  you  think  would  be  necessary  before 
you  are  ready  to  use  DaViTo  in  the  field  for  analysis?” 


Figure  18.  Graph  of  average  response  to  common  question  two.  Note  that  the  y-axis 
correlates  to  the  scale  in  Table  19. 


The  final  common  question  asked  the  subjects  to  list  their  likes  and  dislikes  about 
the  DaViTo  program  for  that  session.  Unlike  the  final  survey  where  questions  were 
written  with  specific  regard  to  the  usability  attributes  described  in  Table  1,  these 
questions  have  no  content  to  lead  the  subject  and  are  strictly  free-form  entry.  The  results 
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of  common  question  three  are  shown  in  Tables  20,  21,  and  22  for  sessions  one,  two,  and 
three,  respectively.  Full  transcription  of  comments  can  be  found  in  Appendix  D. 
Common  liked  features  across  all  three  sessions  were  ease  of  use  and  the  user  interface 
layout.  Navigation  functionality,  simplicity,  and  geo-data  representation  appeared  in  the 
top  five  in  more  than  one  session.  The  single  common  disliked  feature  across  all  three 
sessions  was  response  time.  Mouse  interaction,  lack  of  a  progress  status  notification,  and 
graph  functionality  were  listed  in  the  top  five  on  more  than  one  occasion.  It  is  also  of 
note  that  some  features  were  liked  by  some  participants  while  disliked  by  others. 
Recommendations  for  improvement  based  on  the  liked  and  disliked  features  are 
discussed  in  Chapter  V. 


Top  5  Liked  Features 

Times 

Cited 

Top  5  Disliked  Features 

Times 

Cited 

Ease  of  Use 

24 

Response  Time 

20 

Layout 

14 

Distance  Function 

10 

Navigation  Functionality 

8 

Layout 

6 

Simplicity 

7 

Map  Overlays 

6 

Data  Sets 

6 

Navigation  Function 

4 

Table  20.  Summary  statistics  of  the  top  five  likes  and  dislikes  for  session  one,  common 
question  three. 


Top  5  Liked  Features 

Times 

Cited 

Top  5  Disliked  Features 

Times 

Cited 

Geo-Data  Representation 

6 

Mouse  Interaction 

7 

Ease  of  Use 

5 

Response  Time 

7 

Functionality 

3 

No  Progress  Status 

3 

Layout 

2 

Creating  a  Graph 

2 

Graph  Functionality 

1 

Graph  Functionality 

2 

Table  21.  Summary  statistics  of  the  top  five  likes  and  dislikes  for  session  two,  common 
question  three. 
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Top  5  Liked  Features 

Times 

Cited 

Top  5  Disliked  Features 

Times 

Cited 

Ease  of  Use 

4 

Response  Time 

14 

Geo-Data  Representation 

3 

No  Progress  Status 

9 

Navigation  Functionality 

2 

Graph  Functionality 

9 

Layout 

2 

Map  Labels 

3 

Simplicity 

1 

Mouse  Interaction 

3 

Table  22.  Summary  statistics  of  the  top  five  likes  and  dislikes  for  session  three, 
common  question  three. 


2.  Analysis  of  the  Final  Survey 

The  final  survey  was  written  utilizing  the  five  usability  attributes  described  in 
Table  1,  from  the  Handbook  of  Usability  Testing  (Rubin  &  Chisnell,  2008).  Topics  of 
improvements  and  visualization  were  also  used  to  formulate  questions.  All  questions, 
analysis,  and  responses  are  contained  in  Appendix  D. 

1.  Usefulness 

The  attribute  of  usefulness  refers  to  the  ability  of  a  user  to  achieve  their 
goals  and  their  willingness  to  use  the  product.  If  this  attribute  is  not  considered  in  the 
development  process  designers  may  create  software  from  their  point  of  view  rather  than 
that  of  the  user,  resulting  in  a  deficient  design  (Rubin  &  Chisnell,  2008). 

Questions  concerning  usefulness,  shown  in  Table  23,  all  had  a  score 
greater  than  three  except  Use_6,  which  scored  a  2.66.  It  is  of  note  that  the  user’s  manual 
was  not  consulted  by  any  subject  despite  being  available,  and  there  is  no  electronic  user’s 
manual  or  help  function  accessible  from  the  DaViTo  interface.  While  the  majority  of 
subjects  selected  response  three  (neither  agree  nor  disagree),  the  other  users  based  their 
response  on  the  perceived  difficulty  of  the  software  rather  than  by  actually  referencing 
the  manual.  This  indicates  a  potential  deficiency  in  usefulness  with  regards  to  the  ability 
to  get  assistance  from  a  manual  or  help  tab  within  the  interface.  Question  Use_2  results 
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also  identify  usefulness  deficiencies  in  the  areas  of  graph  interpretation  and  construction 
as  well  as  polygon  construction.  Deficiencies  will  be  addressed  by  recommendations  in 
Chapter  V. 


Usefulness  Question 

Question 

Designator 

Scale 

Frequency 

Mean 

Mode 

Median 

1 

1 

The  tasks  were  easy 
to  complete? 

2 

1 

Use_l 

3 

5 

3.71 

4 

4 

4 

14 

5 

3 

The  terminology  for 
buttons  and 
commands  was  clear 
and  understandable? 

1 

0 

2 

4 

Use_3 

3 

2 

3.79 

4 

4 

4 

13 

5 

5 

A  computer  novice 
would  be  able  to  use 

1 

1 

2 

7 

this  software  with 

Use_4 

3 

4 

3.16 

4 

3 

the  training 
provided? 

4 

11 

5 

1 

A  computer  novice 
would  be  able  to  use 

1 

4 

2 

5 

this  software  with 

Use_6 

3 

11 

2.66 

3 

3 

just  the  user’s 

4 

3 

manual? 

5 

1 

What  was  the  most 

Reading/interpreting  the  graphs. 

difficult  task  to 

Use_2 

Creating  the  graphs. 

complete? 

Drawing  polygons. 

Table  23.  Table  of  usefulness  questions  and  associated  statistics. 
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2.  Effectiveness 


The  attribute  of  effectiveness  refers  to  whether  the  product  behaves  as  the 
user  expects  or  that  users  can  use  the  software  to  do  what  they  desire  it  to  do.  While  this 
can  be  measured  against  a  baseline  rate  of  error,  these  questions  addressed  the  users’ 
perceived  effectiveness  of  their  interaction  with  DaViTo  (Rubin  &  Chisnell,  2008). 

Questions  concerning  effectiveness,  listed  in  Table  24,  all  had  a  score  of 
three  or  greater  except  Effect_4,  which  scored  a  1.875.  The  results  of  this  question  once 
again  address  the  latency  of  the  program  and  point  to  an  effectiveness  deficiency.  This 
can  be  a  source  of  user  frustration,  also  affecting  usefulness  if  users  are  driven  to  stop 
using  the  software.  Question  Effect_3  also  identifies  effectiveness  deficiencies  in  the 
areas  of  software  lag,  the  lack  of  a  processing  status  bar,  and  the  need  for  geographic 
labels  within  the  Map  and  Data  Display  region.  Deficiencies  will  be  addressed  by 
recommendations  in  Chapter  V. 
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Effectiveness 

Question 

Question 

Designator 

Scale 

Frequency 

Mean 

Mode 

Median 

I  understood  where  I 
was  at  in  the 
software  as  I 
progressed  through 
a  task? 

Effect  1 

1 

1 

3.375 

4 

4 

2 

6 

3 

3 

4 

11 

5 

3 

The  mouse 
interacted  with  the 
software  as  I 
expected  it  to? 

Effect  2 

1 

1 

3.667 

4 

4 

2 

4 

3 

2 

4 

12 

5 

5 

I  was  satisfied  with 
the  reaction  time  of 
the  software? 

Effect  4 

1 

11 

1.875 

1 

2 

2 

9 

3 

1 

4 

2 

5 

1 

I  felt  there  was  an 
unnecessary  amount 
of  mouse  clicking  to 
complete  the  given 
task? 

Effect  5 

1 

3 

3.25 

4 

3 

2 

8 

3 

7 

4 

4 

5 

2 

I  made  frequent 
errors  trying  to 
complete  the  given 
task? 

Effect  6 

1 

2 

3 

4 

3 

2 

9 

3 

5 

4 

3 

5 

5 

Was  the  software 
simple  to  use? 
Please  explain. 

Effect  3 

Processing  lag  or  crash. 

Lack  of  a  processing  status  bar. 

Lack  of  geographic  labels. 

Table  24.  Table  of  effectiveness  questions  and  associated  statistics. 
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3.  Efficiency 


The  attribute  of  efficiency  concerns  the  speed  with  which  the  user  can 
accomplish  a  goal(s).  While  usually  measured  against  a  baseline  time,  these  questions 
address  the  user’s  perceived  effective  interaction  with  DaViTo  (Rubin  &  Chisnell,  2008). 

Questions  concerning  efficiency,  listed  in  Table  25,  all  had  a  score  of  three 
or  greater  except  Eff_8  which  scored  a  2.25.  Eff_8  directly  pinpoints  a  serious  lack  of 
efficiency  experienced  by  multiple  users  due  to  software  freezes  and  subsequent  crashes. 
This  can  also  be  a  source  of  user  frustration  affecting  usefulness  if  users  are  driven  to 
stop  using  the  software.  Question  Eff_4  results  also  identify  efficiency  failures  in  visual 
representation  as  a  result  of  no  geographic  labeling  of  countries  or  regions,  and  the 
inability  within  DaViTo  to  sort  different  data  types  within  a  data  set  and  display  their 
data  points  with  different  colors.  This  question  also  had  multiple  responses  again 
identifying  difficulty  reading  graphs  in  DaViTo.  Deficiencies  will  be  addressed  by 
recommendations  in  Chapter  V. 
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Efficiency  Question 

Question 

Designator 

Scale 

Frequency 

Mean 

Mode 

Median 

The  organization  of 
the  screen  matched 
my  expectation  of 
button  location? 

1 

1 

2 

4 

Effl 

3 

5 

3.5 

4 

4 

4 

10 

5 

4 

1 

0 

Similar  icons  and 

2 

3 

activities  are 

Eff_2 

3 

3 

3.83 

4 

4 

consistently  placed? 

4 

13 

5 

5 

1 

0 

Navigation  of 

2 

5 

DaViTo  was 

Eff_3 

3 

3 

3.58 

4 

4 

intuitive? 

4 

13 

5 

3 

1 

0 

The  color  scheme  for 

2 

3 

the  software  was 

Eff_5 

3 

4 

3.79 

4 

4 

appropriate? 

4 

12 

5 

5 

The  text  within 

1 

0 

output  graphical 

2 

8 

representations  was 

Eff_6 

3 

6 

3.21 

2 

3 

easy  to  read  and 

4 

7 

match  to  data? 

5 

3 

The  software  header 
bars  and  dropdown 
lists  made 
navigational  sense? 

1 

2 

2 

0 

Eff_7 

3 

2 

3.83 

4 

4 

4 

16 

5 

4 

1 

9 

The  software  was 
stable  -  no  freezing? 

2 

6 

Eff_8 

3 

3 

2.25 

1 

2 

4 

6 

5 

0 

Was  the  visual 

Lack  of  geographic  labels. 

representation  easy 

Eff_4 

Charts  difficult  to  read. 

to  understand? 

Unable  to  sort  data  types  within  a  set  by  color. 

Table  25.  Table  of  efficiency  questions  and  associated  statistics. 
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4.  Learnability 


The  attribute  of  learnability  is  a  sub-attribute  of  effectiveness  and  concerns 
the  user’s  ability  to  operate  the  system,  whether  it  is  after  some  period  of  familiarization 
and/or  training.  Additionally,  this  element  can  refer  to  a  subject’s  ability  to  use  the 
software  effectively  after  some  period  of  inactivity  (Rubin  &  Chisnell,  2008).  Both  of 
these  were  addressed  by  the  learnability  segment  of  the  final  questionairre. 

Questions  concerning  effectiveness  all  had  a  score  of  three  or  greater 
except  Leam_3,  with  a  score  of  2.17,  and  are  presented  in  Table  26.  Similar  to  the 
second  common  question  of  all  three  sessions,  Learn_3  requires  the  participant  to 
evaluate  if  they  are  adequately  trained  to  use  the  software.  Like  common  question  two, 
the  majority  of  subjects  do  think  additional  training  on  DaViTo  is  necessary.  Also  of 
note  is  Learn_2,  which  like  Use_6,  concerns  the  use  of  the  user’s  manual.  The  responses 
to  Learn_2  averaged  three,  and  10  subjects  agreed  or  strongly  agreed,  while  Use_6  only 
scored  a  2.66  and  only  four  subjects  agreed  or  strongly  agreed.  Despite  these  conflicting 
responses  the  fact  remains  that  the  user’s  manual  was  never  consulted  by  any 
participants,  and  there  is  little  help  available  via  the  interface.  These  deficiencies  will  be 
addressed  by  recommendations  in  Chapter  V. 
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Learnability 

Question 

Question 

Designator 

Scale 

Frequency 

Mean 

Mode 

Median 

I  think  most  people 
could  figure  out  how 
to  use  this  software 
with  the  training 
aids  provided? 

Learn  1 

1 

3 

3.54 

4 

4 

2 

3 

3 

0 

4 

14 

5 

4 

I  think  most  people 
could  figure  out  how 
to  use  this  software 
with  just  the  user’s 
manual? 

Learn  2 

1 

3 

3 

4 

3 

2 

5 

3 

6 

4 

9 

5 

1 

No  more  training  is 
needed  to  effectively 
use  this  software? 

Learn  3 

1 

6 

2.167 

2 

2 

2 

11 

3 

4 

4 

3 

5 

0 

If  given  these  tasks 
in  one  month  to 
complete  again-  I 
feel  I  could  execute 
them? 

Learn  4 

1 

5 

3.22 

4 

4 

2 

2 

3 

3 

4 

10 

5 

4 

I  could  show 
someone  who  has  no 
experience  with 
DaViTo  basic 
functionality? 

Learn  5 

1 

1 

3.79 

4 

4 

2 

2 

3 

2 

4 

15 

5 

4 

More  training  aids 
are  needed  for  me  to 
effectively  learn  to 
use  the  software? 

Learn  6 

1 

0 

3.5 

4 

4 

2 

6 

3 

4 

4 

10 

5 

4 

Table  26.  Table  of  learnability  questions  and  associated  statistics. 
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5. 


Satisfaction 


The  attribute  of  satisfaction  is  linked  to  the  users’  perception  of  the 
software.  Typically  users  will  perfonn  better  when  using  a  satisfying  product  that  meets 
their  needs  (Rubin  &  Chisnell,  2008). 

Questions  concerning  satisfaction,  listed  in  Table  27  had  three  results  with 
scores  less  than  three.  These  questions  are  all  related  to  the  participants  self-assessment 
of  their  abilities  to  use  DaViTo  for  basic  analysis  (Sat  l),  complex  tasks  without  the 
user’s  manual  (Sat_3),  and  using  DaViTo  with  no  assistance  (Sat_6.)  Overall,  this  is 
indicative  of  a  deficiency  in  satisfaction  with  their  ability  to  use  the  software  and  is  likely 
related  to  their  current  level  of  training  and  experience.  Again,  the  user’s  manual  was 
never  consulted  by  any  participants,  and  there  is  no  help  available  via  the  interface.  This 
deficiency  will  be  addressed  by  recommendations  in  Chapter  V. 
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Satisfaction 

Question 

Question 

Designator 

Scale 

Frequency 

Mean 

Mode 

Median 

I  feel  confident  in  my 
abilities  to  use 
DaViTo  for  basic 
analysis? 

1 

3 

2 

3 

Satl 

3 

10 

2.958 

3 

3 

4 

8 

5 

0 

I  think  I  can 
successfully  finish 
basic  tasks  without 
the  user's  manual? 

1 

3 

2 

4 

Sat_2 

3 

6 

3.125 

4 

3 

4 

9 

5 

2 

I  think  I  can 
successfully 
complete  complex 

1 

8 

2 

6 

Sat_3 

3 

4 

2.375 

1 

2 

tasks  without  the 
user's  manual? 

4 

5 

5 

1 

I  think  this  software 
would  add  value  to 
missions  I  have 
performed? 

1 

0 

2 

2 

Sat_4 

3 

11 

3.458 

3 

3 

4 

9 

5 

2 

1 

2 

I  would  recommend 

2 

2 

this  software  to  my 

Sat_5 

3 

12 

3.167 

3 

3 

command? 

4 

6 

5 

2 

I  feel  comfortable 
using  DaViTo 
without  any 
assistance? 

1 

3 

2 

8 

Sat_6 

3 

8 

2.67 

3 

3 

4 

4 

5 

1 

Were  you  satisfied 

Software  latency/lag. 

with  the 

Lack  of  help  functionality. 

performance  of  the 
software?  Please 
explain. 

Sat_7 

Lack  of  graphing  options  and  graphs  difficult  to 
interpret. 

Table  27.  Table  of  satisfaction  questions  and  associated  statistics. 
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6. 


Visualizations 


The  visualization  attribute  was  used  to  identify  subject  opinions  with 
regard  to  the  visual  appearance  of  DaViTo  output  and  the  user  interface.  These  trends 
could  then  be  used  to  generate  recommendations  in  addition  to  those  generated  from  the 
five  usability  attributes. 

Questions  concerning  visualization  are  presented  in  Table  28,  and  had  one 
question  with  a  score  below  three.  This  question,  Visual_l,  had  a  score  of  2.375  and 
asked  users  if  the  output  data  could  be  clearly  interpreted.  The  large  number  of  negative 
responses  correlates  with  previous  questions  regarding  the  quality  of  data  output  in 
DaViTo,  and  reinforces  a  deficiency  in  this  area.  Question  Visual_2  results  were  not  new 
comments  and  are  concurrent  with  similar  comments  from  other  areas.  These 
deficiencies  will  be  addressed  by  recommendations  in  Chapter  V. 

Subjects  were  also  questioned  on  their  ability  to  identify  key  icons  used  in 
DaViTo.  Five  of  the  six  questions  were  answered  with  greater  than  70  percent  accuracy. 
The  navigation  icon  (Visual_15)  performed  poorly  with  less  than  half  of  users  able  to 
identify  it.  While  in  general  icon  recognition  was  not  an  issue,  with  the  exception  of  the 
navigation  icon,  there  were  multiple  comments  throughout  the  free-form  portions  of  the 
survey  to  remove  the  requirement  to  change  cursor  functionality  between  the  four  types 
and  instead  add  right  click  functionality  with  options  to  perfonn  all  actions  from  the 
single  cursor  type.  This  will  be  addressed  in  Chapter  V. 


Visualization 

Question 

Question 

Designator 

Scale 

Frequency 

Mean 

Mode 

Median 

The  way  in  which 
output  data  is 
displayed  is  clear  to 
interpret 

1 

7 

2 

6 

Visuall 

3 

8 

2.375 

3 

2 

4 

1 

5 

2 

What  more  would 

More  chart  options  availal 

ble  to  the  user. 

you  like  to  see  in  the 
output  of  this 

Visual_2 

Ability  to  manipulate  the  charts,  such  as  zoom 
and  view  data  values  on  the  chart. 

software? 

More  functionality  to  tailor  the  data. 

Table  28.  Table  of  visualization  questions  and  associated  statistics. 
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Question 

Designator 

Icon 

Options  Selected 

Frequency 

Visual_13 

Pan 

21 

© 

Map  Navigation 

1 

Gestures 

2 

%  Correct 

87.5 

Visual_14 

A 

Distance 

24 

%  Correct 

100 

Visual_15 

Navigation 

11 

Pan 

4 

+T+ 

Map  Navigation 

8 

I  don’t  know 

1 

%  Correct 

45.83 

Visual_16 

M> 

Forward 

Projection 

21 

Pan 

1 

I  don’t  know 

2 

%  Correct 

87.5 

Visual_17 

Map  Navigation 

17 

Navigation 

5 

Pan 

2 

%  Correct 

70.83 

Visual_18 

i 

Drawing 

24 

%  Correct 

100 

Visualization 

Question 


What  does  this 
icon  mean? 


Table  29.  Table  of  visualization  questions  regarding  icons,  and  associated  statistics. 
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7.  Improvements 


The  inclusion  of  the  improvements  section  with  the  other  attributes  was  an 
effort  to  give  the  participants  an  opportunity  to  voice  ways  to  change  DaViTo  in  the 
context  of  specific  questions.  This  prevented  generic  responses  to  general  questions. 

Questions  concerning  improvements  and  some  of  the  results  are  contained 
in  Table  30,  with  complete  data  in  Appendix  D.  The  question  Improvel  asks  about  the 
user’s  manual.  While  most  subjects  answered  neutrally,  there  was  a  full  spectrum  of 
answers  despite  the  manual  never  having  been  consulted.  When  asked  in  Improve_9 
about  their  preferred  graphs,  the  majority  of  users  chose  the  SBF  by  a  large  margin, 
which  is  concurrent  with  it  exhibiting  the  top  perfonnance  overall.  Deficiencies 
identified  by  the  subjects  in  the  free-form  questions  will  be  addressed  by 
recommendations  in  Chapter  V. 
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Improvement 

Question 

Question 

Designator 

Scale 

Frequency 

Mean 

Mode 

Median 

User’s  manual  is 
easily 

understandable? 

Improve  1 

1 

2 

3 

3 

3 

2 

1 

3 

17 

4 

3 

5 

1 

More  training  aids 
are  needed  for  me  to 
effectively  learn  to 
use  the  software? 

Improve  2 

1 

0 

3.5 

4 

4 

2 

6 

3 

4 

4 

10 

5 

4 

How  would  you 
improve  DaViTo? 
Please  explain. 

Improve_3 

Correct  the  lag  issues. 

Generally,  improve  the  graphs  and  interaction. 

Add  a  progress  icon. 

What  was  the  most 
frustrating  feature  of 
DaViTo? 

Improve  4 

Lack  of  on-screen  help  available. 

Standardize  mouse  button  functionality. 

General  -  Lag  and  charts. 

What  did  you  most 
like  about  DaViTo? 
Please  explain. 

Improve_5 

Ability  to  display  data  geographically. 

Integrated  data  analysis. 

Simple  and  intuitive  interface. 

What  would  you 
change  about  the 
color  scheme  /  page 
layout?  Please 
explain. 

Improve_6 

Area  relief  or  3D  capability  like  Google  Earth. 

Color  scheme  is  bland,  add  contrast. 

Change  default  data  point  color. 

How  would  you 
improve  the  mouse 
interaction  with  the 
software?  Please 
explain. 

Improve_7 

Right  click  on  map  shows  all  available  options. 

Quicker  response  to  right  clicks. 

Highlighting  or  positive  indication  of  selection. 

How  would  you  like 
to  see  the  software 
employed?  Please 
explain. 

Improve_8 

Multiple,  see  Appendix  D. 

What  graph  did  you 
think  conveyed  the 
information  in 
general  the  best? 

Imrpove_9 

Options 

Frequency 

SBF 

14 

CCF 

2 

SLMF 

6 

LCF 

0 

SAF 

2 

Table  30.  Table  of  improvement  questions  and  associated  statistics. 
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3.  Survey  Results  Summary 


Multiple  usability  deficiencies  were  identified  by  the  output  data  from  both  the 
three  common  questions  for  all  sessions,  as  well  as  the  extensive  session  three  survey. 
Recommendations  for  improvement  will  be  made  in  Chapter  V. 

The  common  questions  were  more  general  than  the  session  three  survey.  The  first 
two  questions  did  not  identify  deficiencies,  rather  were  related  to  the  training  received 
and  subject  opinions  of  additional  training  necessary.  The  responses  will  serve  as  a  basis 
for  improvement  recommendations.  The  final  question  identified  general  likes  and 
dislikes.  The  dislikes  correlate  to  more  specific  comments  made  in  the  free-form 
questions  of  the  final  survey  and  will  be  used  to  formulate  recommendations  for  the  next 
release  of  the  DaViTo  software. 

The  final  survey  was  extensive  and  exhausting.  Multiple  deficiencies  were 
identified  by  both  free-form  response  questions  and  scaled  answer  questions.  These 
deficiencies  will  be  incorporated  into  the  recommendations  in  Chapter  V. 
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V.  RECOMMENDATIONS 


The  main  objective  of  this  thesis  was  to  provide  clear  and  actionable 
recommendations  to  TRAC  Monterey  to  improve  the  usability  of  the  DaViTo  software. 
Data  analysis  was  divided  into  three  distinct  sections  for  analysis  and  results  reporting, 
and  the  recommendations  will  follow  this  same  format.  Identical  recommendations 
supported  with  different  bases  will  be  reported  in  the  tables.  Additionally, 
recommendations  contained  in  the  tables  in  this  Chapter  are  connected  to  their  primary 
usability  attribute  in  Table  1.  If  other  attributes  are  related,  they  will  be  presented  in  the 
discussion  paragraphs.  A  comprehensive  table  of  recommendations  can  be  found  in 
Appendix  A. 

A.  RECOMMENDATIONS  BASED  ON  TASK  RESULTS 

Eye-tracking  and  interface  interaction  was  the  primary  source  of  data  for  the  task 
results  analysis  presented  in  Chapter  IV.  Three  data  sources  were  used  to  generate  these 
recommendations:  mouse  clicks,  duration  of  time  to  complete  tasks,  and  heat  maps  of  all 
participants  gaze  locations. 

The  first  two  recommendations  listed  in  Table  31,  A.l  and  A.2,  were  touched  on 

by  nearly  every  subject,  and  will  be  mentioned  in  multiple  sections  of  Chapter  V  for  a 

variety  of  different  bases.  Recommendation  A1  requires  revising  the  software  code  to 

improve  software  processing,  response  time,  and  stability.  Recommendation  A2  is  to 

implement  a  processing  status  bar  to  alert  the  user  that  the  software  is  operating  and  is  not 

frozen.  As  discussed  in  Chapter  IV,  multiple  mouse  clicks  would  take  place  and  wasted 

time  elapse  as  participants  waited  with  growing  impatience  for  the  polygon  to  form  and 

populate.  This  lack  of  efficiency  in  the  software  code  resulted  in  user  frustration,  which 

the  absence  of,  is  one  of  the  definitions  of  usability  mentioned  in  Chapter  II.  Not  only  is 

efficiency  at  issue  here  but  also  usefulness  and  satisfaction.  As  DaViTo  performance 

deficiencies  reduce  the  ability  of  the  user  to  complete  their  goals  efficiently,  the  user  then 

becomes  less  likely  to  use  the  product  at  all,  and  their  perception  of  the  product  becomes 

more  negative  (Rubin  &  Chisnell,  2008).  Finally,  most  software  has  lag  time  at  some 
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point  as  it  computes  things,  and  the  implementation  of  A. 2  would  reduce  the 
dissatisfaction  of  users.  As  the  moments  pass  at  least  the  user  will  know  the  program  is 
processing  rather  than  being  left  guessing  as  to  whether  it  is  simply  frozen. 

The  excess  mouse  clicks  were  the  result  of  frustration  and  no  status  of  progress; 
however,  the  mouse  clicks  should  not  be  necessary  at  all.  The  procedure  of  drawing  the 
shape  and  then  right-clicking  outside  the  shape  to  initiate  its  formation  is  unnecessary  and 
counter-intuitive.  Like  other  programs  with  shape  drawing  functionality  (Microsoft 
Paint,  PowerPoint,  Word,  etc.),  the  code  should  be  modified  to  not  require  this  additional 
click  as  recommended  by  A. 3,  but  rather  initiate  once  the  mouse  button  is  released. 

Finally,  the  heat  map  analysis  indicated  a  large  concentration  of  subjects  gazing 
and  clicking  in  the  Map  Controls  Region.  Again,  the  file  manipulations  functionality  is 
counter-intuitive  to  common  programs  used.  Recommendation  A. 4  will  increase 
effectiveness  for  users. 


Designator 

Recommendation 

Description 

Attribute  and  Basis  for  Recommendation 

A.l 

Improve  software 
processing,  response 
time,  and  stability. 

Efficiency  -  When  drawing  user-defined 
polygons  there  was  excessive  lag  time  or  the 
software  crashed. 

A.2 

Implement  a  progress 
status  bar. 

Effectiveness  -  When  drawing  polygons  the  lag 
time  led  subjects  to  continue  clicking  outside  the 
polygon  because  there  was  no  indication  of 
progress. 

A.3 

Draw  and  populate  a 
polygon  when  drawn 
by  the  user. 

Effectiveness  -  Right  clicking  outside  of  the 
polygon  was  counter-intuitive  to  users. 

A.4 

Implement  file 
operations  in  the  main 
menu  ribbon. 

Effectiveness  -  Heat  mapping  shows  that  many 
users  looked  to  the  menu  ribbon  for  file 
operations. 

Table  3 1 .  List  of  recommendations  and  the  basis  derived  from  analysis  of  the  task 
results. 
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B.  RECOMMENDATIONS  BASED  ON  GRAPH  RESULTS 

Three  of  the  four  development  goals  for  DaViTo  discussed  in  Chapter  II  involve 
graphical  representation  of  data.  The  production  of  useful  visualization  and  data  analysis, 
and  representing  data  with  graphical  output  that  can  be  interpreted  by  ground 
commanders  are  the  root  of  two  of  these  (Evangelista,  Henry,  Buttrey,  &  Whitaker, 
2012).  The  usability  of  DaViTo  is  significantly  deficient  in  this  regard.  In  addition  to  the 
two  recommendations  in  Table  32,  there  are  more  concerning  the  functionality  of  graphs 
in  the  next  section  generated  from  the  survey  data. 

It  was  identified  that  the  default  graphing  option,  Line  Chart  Format  (LCF),  was 
the  lowest  scoring  option  overall  in  the  three  sessions.  Fortunately  the  highest  perfonner, 
Stacked  Bar  Format  (SBF),  is  the  one  and  only  other  option  directly  on  the  DaViTo 
interface.  However,  this  requires  more  from  the  user,  such  as  knowing  there  is  another 
option  and  realizing  that  this  option  may  better  convey  the  data.  Implementing  B.l  will 
make  the  SBF  the  default  graph  type.  This  will  have  an  immediate  impact  on  the 
efficiency  of  the  software  by  giving  the  user  the  best  chance  at  successfully  interpreting 
the  graph.  This  will  also  improve  the  usefulness,  effectiveness,  and  satisfaction  attributes 
of  DaViTo. 

The  second  recommendation,  B.2,  will  add  more  options  for  different  graph  types 
and  further  improve  the  usefulness  of  DaViTo.  It  is  clear  from  the  results  that  while  the 
SBF  is  the  top  overall  perfonner,  different  graphs  produce  better  results  dependent  upon 
the  question.  Similarly,  most  software  with  graphing  functionality  (Excel,  JMP)  are  not 
limited  to  two  graph  types,  but  give  their  users  a  plethora  of  choices,  as  well  as  provide 
brief  explanations  of  the  strengths  of  each  format.  Admittedly,  DaViTo  has  the 
capability  to  produce  different  graph  types  either  through  the  use  of  R  or  importing  the 
data  to  Excel.  Unfortunately,  this  is  laborious  and  counter  to  one  of  the  goals  stated  for 
the  development  of  DaViTo  which  was  that  the  software  would  be  easy  to  use  for 
someone  unfamiliar  with  statistical  methods  (Evangelista,  Henry,  Buttrey,  &  Whitaker, 
2012).  Requiring  a  user  that  is  unfamiliar  with  statistical  methods  such  as  those 
employed  by  R,  or  requiring  the  importation,  manipulation,  and  eventual  graphing  of  data 
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in  Excel,  is  not  realistic.  Including  more  options  for  graph  types  in  the  user  interface 
such  that  they  are  easily  accesible  will  better  meet  the  original  development  goals  of 
DaViTo. 


Designator 

Recommendation 

Description 

Attribute  and  Basis  for  Recommendation 

B.l 

Make  the  Stacked  Bar 
Format  (SBF)  the 
default  graph  type. 

Efficiency  -  The  SBF  graph  was  the  most 
conducive  to  generating  a  correct  answer  from 
experiment  subjects. 

B.2 

Add  more  options  for 
different  graph  types. 

Usefulness  -  Different  graphs  have  better 
performance  relative  to  the  question  asked. 

Table  32.  List  of  recommendations  and  the  basis  derived  from  analysis  of  the  graph 
results. 


C.  RECOMMENDATIONS  BASED  ON  SURVEY  RESULTS 

Survey  results  constituted  the  only  source  of  participant  feedback  on  DaViTo. 
Although  subjective,  the  majority  of  participants  seemed  genuinely  interested  in 
providing  useful  feedback,  ensuring  the  hours  they  devoted  without  compensation  to  this 
research  was  not  wasted. 

1.  Recommendations  Based  on  the  Three  Common  Questions 

Currently,  the  only  training  product  developed  for  and  distributed  with  DaViTo  is 
a  26  slide  PowerPoint  presentation.  While  this  covers  much  of  the  basic  functionality  of 
the  software,  it  is  not  effective  at  training  a  user  to  use  the  software  such  that  they  meet 
the  goals  stated  when  developing  DaViTo.  Implementing  recommendations  C.l  and  C.2, 
listed  in  Table  33,  would  address  the  current  learnability  deficiency  in  the  software. 
Recommendation  Cl  is  to  develop  a  tutorial  based  training  product  for  DaViTo.  While 
the  tutorial  method  developed  for  this  experiment  was  not  itself  evaluated  against  the 
PowerPoint  presentation  or  any  other  training  methods,  users  did  rate  these  sessions 
highly  on  exit  surveys,  with  an  average  of  4.55  on  a  scale  of  one  to  five,  and  21  of  24 
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users  were  able  to  complete  all  final  session  tasks,  albeit  with  only  one  perfect  score. 
Recommendation  C2  is  to  distribute  an  in  depth  training  product  with  DaViTo.  A 
thoroughly  prepared  and  executed  training  plan  could  teach  the  basic  use  of  DaViTo  in 
several  hours  of  training.  Final  responses  after  session  three  indicated  that  users 
estimated  they  would  require  five  to  seven  total  hours  of  training  to  feel  comfortable 
using  DaViTo  in  the  field,  and  50  percent  of  users  agreed  that  a  novice  could  use  the 
software  with  just  the  training  provided. 


Designator 

Recommendation 

Description 

Attribute  and  Basis  for  Recommendation 

C.l 

Develop  a  tutorial 
based  training  product 
for  DaViTo. 

Leamability  -  Currently  there  are  no  training 
products  for  DaViTo  beyond  a  basic  PowerPoint 
slide  show. 

C.2 

Distribute  a  training 
product  with  DaViTo. 

Leamability  -  There  is  currently  a  PowerPoint 
slide  show  distributed  with  DaViTo. 

Table  33.  List  of  recommendations  and  the  basis  derived  from  analysis  of  the  three 
common  question  survey  results. 


2.  Recommendations  Based  on  Survey  Usefulness  Questions 

Designers  did  design  a  Help  tab  on  the  DaViTo  menu  ribbon;  however,  it 
currently  has  no  functionality  beyond  providing  a  link  to  the  OpenMap  website  help. 
There  is  no  DaViTo  specific  help  and  the  user’s  manual  is  not  available  from  within  the 
software.  Linking  the  user’s  manual  to  the  help  tab  on  the  ribbon  bar  to  implement 
recommendation  C.3  in  Table  34,  would  correct  this  deficiency.  Without  even  seeing  the 
user’s  manual,  83  percent  of  subjects  were  either  neutral  or  disagreed  that  a  computer 
novice  would  be  able  to  use  DaViTo  with  only  the  user’s  manual.  Despite  this, 
accessibility  to  the  user’s  manual  would  increase  usefulness  by  providing  a  reference  for 
assistance  when  problems  arise,  and  linking  it  to  the  help  tab  would  place  it  where  most 
users  would  expect  as  it  is  commensurate  with  commonly  used  software. 
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When  subjects  were  asked  if  the  software  is  simple  to  use,  it  was  most  frequently 
stated  that  the  three  most  difficult  tasks  to  accomplish  involved  graph  interpretation, 
graph  creation,  and  generating  user-defined  polygons.  Graph  tools  such  as  a  scrolling 
capability,  and  mouse-over  functionality  to  display  the  value  of  the  data  point  would 
assist  in  the  implementation  of  recommendation  C.4,  which  is  to  include  graph  tools  to 
improve  graph  readability.  C.5  recommends  making  graphs  easier  to  create.  This  is 
likely  related  to  the  excessive  mouse  clicking  required  when  drawing  a  polygon  and  the 
subsequent  graph  creation.  Finally,  drawing  and  populating  a  polygon  when  drawn  by 
the  user,  designated  recommendation  C.6,  is  identical  to  A. 3,  but  is  based  on  a  usefulness 
deficiency  identified  by  the  participants. 


Designator 

Recommendation 

Description 

Attribute  and  Basis  for  Recommendation 

C.3 

Link  the  user’s  manual 
to  the  Help  tab  on  the 
ribbon  bar. 

Usefulness  -  Currently  the  user’s  manual  is  not 
accessible  via  the  DaViTo  interface. 

C.4 

Include  graph  tools  to 
improve  graph 
readability. 

Usefulness  -  Graph  interpretation  was  identified 
by  subjects  as  the  one  of  the  most  difficult  tasks. 

C.5 

Make  graphs  easier  to 

create. 

Usefulness  -  Graph  creation  was  identified  by 
subjects  as  the  one  of  the  most  difficult  tasks. 

C.6 

Draw  and  populate  a 
polygon  when  drawn 
by  the  user. 

Usefulness  -  Polygon  functions  were  identified 
by  subjects  as  one  of  the  most  difficult  tasks. 

Table  34.  List  of  recommendations  and  the  basis  derived  from  analysis  of  the 
usefulness  portion  of  the  survey  results. 
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3. 


Recommendations  Based  on  Effectiveness  Questions 


Recommendations  C.7  and  C.8,  are  both  recommendations  to  improve  software 
processing  latency  and  stability.  This  recommendation  was  previously  identified  with 
recommendation  A. 3.  Similarly  recommendation  C.9,  to  implement  a  progress  status  bar, 
is  identical  to  A. 2. 

The  Map  and  Data  Display  Area  contains  legal  boundaries  for  countries,  but  there 
are  no  labels.  This  was  identified  by  subjects  as  a  deficiency  when  using  DaViTo  in  the 
final  survey.  Recommendation  C.10  in  Table  35  addresses  this  deficiency  by 
recommending  the  implementation  of  geographic  labels  in  the  Map  and  Data  Display 
Area.  Specifically,  in  the  final  session  participants  were  directed  to  find  Kenya  and  load 
a  Shapefile.  This  required  geographic  knowledge  of  Africa  not  necessary  when  using  a 
typical  map  with  appropriate  labeling. 


Designator 

Recommendation 

Description 

Attribute  and  Basis  for  Recommendation 

C.7 

Improve  the  software 
reaction  time. 

Effectiveness  -  83  percent  of  subjects  were  not 
satisfied  with  the  software  reaction  time. 

C.8 

Improve  software 
processing,  response 
time,  and  stability. 

Effectiveness  -  Subjects  stated  this  as  the  worst 
deficiency  that  reduced  the  simplicity  of  use  of 
DaViTo. 

C.9 

Implement  a  progress 
status  bar. 

Effectiveness  -  Subjects  stated  this  as  the  second 
worst  deficiency  that  reduced  the  simplicity  of 
use  of  DaViTo. 

C.10 

Include  geographic 
labels  in  the  Map  and 
Data  Display  Area. 

Effectiveness  -  Subjects  stated  this  as  the  third 
worst  deficiency  that  reduced  the  simplicity  of 
use  of  DaViTo. 

Table  35.  List  of  recommendations  and  the  basis  derived  from  analysis  of  the 
effectiveness  portion  of  the  survey  results. 
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4. 


Recommendations  Based  on  Efficiency  Questions 


Recommendation  C.ll,  in  Table  36,  is  again  related  to  improving  software 
processing  latency  and  is  the  same  as  recommendation  A. 3.  Similarly,  recommendation 
C.12,  including  geographic  labels,  is  identical  to  C.10. 

Recommendation  C.13  will  allow  data  sets  of  different  data  types  to  be  sorted 
with  different  colors.  Large  datasets  such  as  the  one  used  frequently  in  the  experiment, 
WITS. csv,  have  multiple  data  types.  Additionally,  when  multiple  datasets  are  loaded, 
they  all  continue  to  use  the  default  gray  color  scheme.  Automatic  color  coding  of  the 
different  subsets  was  recommended  by  multiple  subjects  to  improve  the  visual 
representation  of  data  on  the  Map  and  Data  Display  Area. 


Designator 

Recommendation 

Description 

Attribute  and  Basis  for  Recommendation 

C.ll 

Improve  software 
processing,  response 
time,  and  stability. 

Efficiency  -  67  percent  of  subjects  disagreed  that 
the  software  was  stable  and  not  prone  to  crashing. 

C.12 

Include  geographic 
labels  in  the  Map  and 
Data  Display  Area. 

Efficiency  -  Subjects  reported  this  as  the  top 
deficiency  that  made  visual  representations  easy 
to  read. 

C.13 

Make  data  types 
within  a  data  set 
sortable  by  color. 

Efficiency  -  Subjects  reported  this  as  a  deficiency 
that  made  visual  representations  difficult  to  read. 

Table  36.  List  of  recommendations  and  the  basis  derived  from  analysis  of  the  efficiency 
portion  of  the  survey  results. 


5.  Recommendations  based  on  Learnability  Questions 

Most  questions  in  the  learnability  segment  of  the  final  survey  reflect  positively 
upon  the  training  developed  for  the  experiment.  The  majority  of  subjects  believe  that 
with  this  training  most  people  could  use  DaViTo,  they  could  complete  the  final  session 
tasks  again  in  one  month,  and  that  they  could  teach  a  user  the  basic  functionality  of 
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DaViTo.  Despite  this,  a  majority  also  believe  more  training  products  are  needed. 
Because  of  these  results,  Table  37  lists  recommendation  C.14  (identical  to  C.l)  to 
endorse  a  tutorial  based  training  program  and  C.l 5  (identical  to  C.2)  recommends  that  it 
is  distributed  with  DaViTo.  Training  should  be  developed  with  the  DaViTo  software 
objective  in  mind  that  a  user  whom  is  unfamiliar  with  statistical  methods  and  Geographic 
Infonnation  Systems  could  use  the  software.  This  will  reinforce  the  other  software  goals 
of  users  that  can  produce  useful  visualizations  that  can  be  grasped  and  understood  by 
ground  commanders  (Evangelista,  Henry,  Buttrey,  &  Whitaker,  2012). 


Designator 

Recommendation 

Description 

Attribute  and  Basis  for  Recommendation 

C.14 

Develop  a  tutorial 
based  training  product 
for  DaViTo. 

Leamability  -  75  percent  of  subjects  reported  that 
most  people  could  figure  out  how  to  use  DaViTo 
with  the  tools  provided;  however  58  percent 
believe  more  training  products  are  needed. 

C.15 

Distribute  a  training 
product  with  DaViTo. 

Leamability  -  88  percent  of  subjects  are  either 
neutral  or  believe  that  more  training  is  necessary 
to  effectively  be  able  to  use  DaViTo. 

Table  37.  List  of  recommendations  and  the  basis  derived  from  analysis  of  the 
leamability  portion  of  the  survey  results. 


6.  Recommendations  Based  on  Satisfaction  Questions 

The  satisfaction  segment  of  the  final  survey  generated  only  one  unique 
recommendation,  found  in  Table  38.  Subjects  desired  that  a  help  capability  be  designed 
into  the  DaViTo  interface.  Currently  there  is  little  help  functionality,  only  a  link  to  the 
OpenMap  developer’s  page,  and  no  user’s  manual  is  accessible  from  the  interface. 
Recommendation  C.16,  to  link  the  user’s  manual  to  the  help  tab  is  a  repeat  of  C.3.  If  the 
user’s  manual  provides  proper  guidance,  this  will  increase  user  satisfaction.  Similarly, 
recommendation  C.17  recommends  going  further  with  a  comprehensive  help  capability. 
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This  feature  would  put  DaViTo  on  par  with  more  familiar  software  packages  that  utilize 
searchable  help  databases  in  addition  to  online  help  capabilities. 

Additional  recommendations  in  Table  38  are  C.18  and  C.19,  call  for  improved 
software  stability,  and  more  graphing  tools,  respectively.  These  are  repeats  of  A.l  and 
C.4,  respectively.  Both  recommendations  were  derived  from  participant  free-form 
responses  to  satisfaction  questions. 


Designator 

Recommendation 

Description 

Attribute  and  Basis  for  Recommendation 

C.16 

Link  the  user’s  manual 
to  the  Help  tab  on  the 
ribbon  bar. 

Satisfaction  -  46  percent  of  users  think  they  can 
conduct  basic  tasks,  and  25  percent  of  users  think 
they  can  complete  complex  tasks,  without  the 
user’s  manual. 

C.17 

Develop  a 
comprehensive  help 
capability  for  DaViTo. 

Satisfaction  -  Only  21  percent  of  subjects  are 
comfortable  using  DaViTo  without  assistance. 
Additionally  this  was  the  second  most  frequent 
complaint  with  regards  to  satisfaction. 

C.18 

Improve  software 
processing,  response 
time,  and  stability. 

Satisfaction  -  This  was  the  top  complaint  with 
regards  to  satisfaction. 

C.19 

Include  graph  tools  to 
improve  graph 
readability. 

Satisfaction  -  This  was  the  third  most  frequent 
complaint  with  regards  to  satisfaction. 

Table  38.  List  of  recommendations  and  the  basis  derived  from  analysis  of  the 
satisfaction  portion  of  the  survey  results. 


7.  Recommendations  Based  on  Visualization  Questions 

The  visualization  segment  of  the  final  survey  generated  only  one  unique 
recommendation,  found  in  Table  39.  There  was  some  confusion  among  the  users  with 
regard  to  the  identification  of  the  icons  discussed  in  recommendation  C.23.  Since  both 
have  similar  functionality,  it  is  recommended  to  use  a  common  icon. 
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Additional  recommendations  in  Table  39  are  C.20,  C.21,  and  C.22.  These  are 
repeats  of  C.4,  B.2,  and  C.13,  respectively.  These  recommendations  were  derived  from 
participant  free-fonn  responses  to  visualization  questions  and  are  tailored  to  improve  the 
interpretability  of  the  displayed  output  data  within  DaViTo. 


Designator 

Recommendation 

Description 

Attribute  and  Basis  for  Recommendation 

C.20 

Include  graph  tools  to 
improve  graph 
readability. 

Visualization  -  Only  12.5  percent  of  subjects 
agreed  that  output  data  is  clear  to  interpret,  and 
the  second  most  frequent  response  to  desired 
changes  to  the  output  was  more  graph 
manipulation  tools. 

C.21 

Add  more  options  for 
different  graph  types. 

Visualization  -  The  most  frequent  response  to 
desired  changes  to  software  output  was  more 
graph  options. 

C.22 

Make  data  types 
within  a  data  set 
sortable  by  color. 

Visualization  -  Only  12.5  percent  of  subjects 
agreed  that  displayed  output  data  is  clear  to 
interpret,  and  the  third  most  frequent  response  to 
desired  changes  to  the  output  was  a  greater 
capability  to  tailor  the  data. 

C.23 

Alter  the  cursor 
navigation  button 

+1+ 

to  match  the  map 
navigation  icon. 

Rta 

tss 

Visualization  -  54  percent  of  subjects  could  not 
correctly  identify  the  cursor  navigation  icon.  7 1 
percent  of  subjects  could  correctly  identify  the 
map  navigation  icon.  Some  incorrect  answers  for 
each  were  the  other  icons  title.  The  functionality 
associated  with  these  icons  is  very  similar. 

Table  39.  List  of  recommendations  and  the  basis  derived  from  analysis  of  the 
visualization  portion  of  the  survey  results. 
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8. 


Recommendations  Based  on  Improvement  Questions 


The  final  segment  of  the  session  three  survey  asked  a  series  of  questions  as  to 
how  the  subjects  would  improve  DaViTo.  Many  of  the  recommendations  are  repeats  of 
previous  comments  and  were  mentioned  earlier  with  regards  to  specific  usability 
attributes,  so  only  new  recommendations  are  discussed  in  Table  40. 

Recommendation  C.24  was  mentioned  by  multiple  users,  as  a  desire  for  standard 
two  button  mouse  functionality.  Software  designed  for  Microsoft  Windows  based 
systems  is  typically  developed  in  anticipation  of  a  two  or  three  button  scrolling  mouse 
being  the  interaction  device  of  choice.  In  most  cases,  left  clicking  selects  an  object  or 
carries  out  a  function;  right  clicking  brings  up  a  menu  of  options.  In  DaViTo  a  specific 
converse  to  this  was  right  clicking  outside  of  a  polygon  in  order  for  it  to  be  drawn  and 
populated  with  statistics.  This  additional  click  was  problematic  and  confusing  for  some 
participants. 

Today  applications  such  as  Google  Earth,  MapQuest,  and  other  mapping  tools 
have  a  rich  user  interface.  Not  only  are  they  more  visually  appealing,  it  also  has 
functionality  when  trying  to  understand  the  terrain  of  the  area  you  are  looking  at,  and 
keeps  users  more  engaged.  Recommendations  C.25,  C.26,  and  C.27  in  Table  40  will 
incorporate  a  3D  mapping  capability,  add  contrast  to  the  color  scheme,  and  change  the 
default  data  point  color  from  gray  to  a  color  with  increased  contrast.  This  will  improve 
the  visualization  of  DaViTo,  as  well  as  its  usefulness  and  satisfaction. 
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Designator 

Recommendation 

Description 

Attribute  and  Basis  for  Recommendation 

C.24 

Standardize  two 

button  mouse 
functionality. 

Multiple  Attributes  -  Follow  the  standard 
functionality  for  a  two-button  mouse  that  most 
software  developers  utilize. 

C.25 

Incorporate  an  area 
relief  option  or  3D 
capability  similar  to 
Google  Earth. 

Visualization  -  Top  response  when  subjects  were 
asked  what  they  would  change  about  the  color 
scheme  or  user  interface  layout. 

C.26 

The  color  scheme  is 

bland.  Add  more 

contrast. 

Visualization  -  In  an  effort  to  look  more  inviting 
to  users  software  should  have  visual  appeal,  even 
in  military  applications. 

C.27 

Change  the  default 
data  point  color. 

Visualization  -  The  current  default  of  gray  has  no 
contrast  with  the  interface  due  to  frequent  use  of 
different  shades  of  gray. 

Table  40.  List  of  recommendations  and  the  basis  derived  from  analysis  of  the 
improvement  portion  of  the  survey  results. 


D.  ADDITIONAL  AUTHOR  RECOMMENDATIONS 

Throughout  the  development  of  the  experiment  and  observing  subject  sessions  the 
authors  of  this  thesis  received  numerous  hours  of  exposure  to  DaViTo.  Table  41  contains 
additional  recommendations  by  the  authors  based  on  their  observations  and  interaction. 

Recommendation  D.l  would  increase  the  usable  area  of  the  software  for  data 
representation  and  analysis  by  allowing  users  to  collapse  the  menu  areas  on  the  left  of  the 
user  interface.  Once  the  files  are  opened  or  the  options  in  the  segments  of  the  interface 
on  the  left  side  of  the  screen  are  complete,  they  are  no  longer  very  useful  and  only 
remove  work  space  from  the  user.  D.2  and  D.3  are  intended  to  address  future 
possibilities  for  the  use  of  DaViTo  as  discussed  with  TRAC  Monterey.  It  is  desirable  for 
the  use  of  this  software  to  spread  beyond  the  realm  of  the  small  group  of  analysts  it  was 
designed  for,  and  the  inclusion  of  nautical  capabilities  would  advance  this.  Training  for 
the  new  target  audience  as  well  as  advanced  training  for  the  initial  audience  would 
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increase  usefulness  for  both  parties.  To  get  the  tool  in  use  beyond  the  anny, 
incorporation  of  mapping  features  that  benefit  the  other  service  branches  would  be 
helpful. 

Finally,  recommendation  D.4  addresses  the  development  of  a  database  for  use 
with  DaViTo  as  well  as  standardized  data  formats.  The  output  of  DaViTo  is  only  as  good 
as  the  infonnation  entered  into  it.  It  was  stated  in  Data  Visualization  Tool  that  there  are 
vast  amounts  of  data  that  are  not  being  properly  analyzed  and  the  subsequent  results 
getting  to  decision  makers  (Evangelista,  Henry,  Buttrey,  &  Whitaker,  2012).  Through 
conversations  with  TRAC  Monterey  there  does  not  currently  exist  such  a  database.  This 
should  be  developed  concurrently  with  DaViTo  to  maximize  its  potential  and  ensure  the 
tool  is  able  to  be  utilized  to  its  fullest  potential. 
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Designator 

Recommendation 

Description 

Basis  for  Recommendation 

D.l 

Add  a  collapse 
capability  to  the  Data 
Sets,  Predefined 
Polygons,  and  Regions 
Selection  List  areas  of 

the  interface 

Visualization  -  This  will  expose  more  of  the  Map 
and  Data  Display  area,  reducing  clutter  and 
allowing  the  user  more  capability  to  work  within 
the  primary  workspace  of  the  interface. 

D.2 

Develop  training 
products  for  both  a 
basic  user  and 
experienced  analyst. 

Usefulness  -  Recommendations  C.l  and  C.2 
address  the  development  of  training  products. 
There  are  complex  statistical  capabilities  in 
DaViTo.  An  analyst  with  advanced  skills  would 
benefit  from  tailored  training  that  goes  beyond  the 
capabilities  in  the  user  interface. 

D.3 

Extend  mapping 
capability  to  include 
nautical  charts. 

Usefulness  -  Many  Navy  subjects  identified 
additional  uses  such  as  counter-piracy  or  sea- 
based  counter-drug  operations. 

D.4 

Develop  a 

standardized  Anny  (or 
service  wide)  database 
and  data  formats  for 

use  with  DaViTo. 

Usability  -  Without  a  formal  database  and  data 
formats  the  tool  itself  may  suffer  reduced 
usability. 

Table  41.  List  of  author  recommendations  and  the  basis  derived  from  the  observations 
of  subjects  and  interaction  with  DaViTo. 
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VI.  CONCLUSION  AND  FUTURE  WORK 


A.  SUMMARY 

The  Data  Visualization  Tool  developed  by  TRAC  Monterey  is  a  powerful  piece  of 
software  that  has  the  potential  to  be  instrumental  in  future  military  operations.  If 
properly  developed  through  continuous  revision  and  improvement,  it  will  see  broader 
employment.  The  tool  itself  has  wide-ranging  application  throughout  the  Army  and 
potentially  the  entire  military. 

This  thesis  was  limited  in  scope  to  the  usability  of  DaViTo,  although  there  are 
other  opportunities  for  research.  A  formal  usability  study  was  not  conducted  during  the 
development  of  DaViTo  and  as  such  this  thesis  was  done  in  lieu  of  that  following  the 
initial  release.  The  primary  objective  was  to  provide  actionable  recommendations  to 
TRAC  Monterey  that  could  be  used  to  improve  the  usability  of  the  tool  through  future 
revisions. 

Usability  has  many  definitions,  but  in  the  context  of  this  thesis  it  can  be  best 
described  as  “the  quality  of  a  system  with  respect  to  ease  of  learning,  ease  of  use,  and 
user  satisfaction”  (Rosson  &  Carroll,  2002).  As  a  tool  for  the  analysis  of  data  with  both 
spatial  and  temporal  components  DaViTo  has  the  potential  to  be  instrumental.  Increasing 
the  usability  of  DaViTo  by  improving  upon  all  of  its  attributes  will  enhance  that 
potential. 

The  experiment  itself  was  developed  to  rapidly  train  and  familiarize  participants 
with  DaViTo.  Once  complete,  gathering  data  from  subjects  was  the  next  step.  With  24 
volunteer  participants  over  70  man-hours  were  spent  training,  testing,  and  surveying 
them.  Data  was  collected  through  eye-tracking  while  conducting  tasks,  inspecting 
graphs,  and  answering  numerous  survey  questions. 

In  order  to  compartmentalize  collected  data  for  analysis  it  was  split  into  three 
groups.  The  first  was  the  collection  of  data  through  eye-tracking.  This  data  was  useful 
for  determining  where  subjects  are  focusing  their  attention  while  trying  to  complete  a 
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task.  The  software  used,  EyeWorks  Record,  also  collected  mouse  clicks  and  tracked  the 
time  elapsed  from  task  to  task.  The  second  method  of  data  collection  was  by  presenting 
participants  with  multiple  graphs,  asking  quesitons  about  them,  and  then  analyzing  their 
responses.  The  outcome  of  this  portion  of  the  thesis  was  useful  at  determining  the  best 
ways  to  present  infonnation  to  decision  makers  in  a  manner  in  which  they  can 
understand.  Finally,  a  huge  amount  of  data  directly  from  the  users  was  collected  in  brief 
questioning  in  the  first  two  sessions,  followed  by  a  complex  and  thorough  final  surery  in 
session  three.  This  data  was  particularly  useful  for  statistical  analysis  of  questions  related 
to  the  usability  attributes  identified  in  Table  1,  as  well  as  the  development  of 
recommendations  directly  from  subject  free-form  responses. 

Methods  to  improve  DaViTo  is  the  heart  of  this  research.  Four  subsets  of 
recommendations  were  generated  from  the  results  of  the  data  analysis.  The  first  subset 
was  derived  from  the  results  of  subject  tasks.  Recommendations  were  developed  that 
would  improve  usability  in  nearly  all  attributes.  The  second  subset  of  recommendations 
were  based  on  the  results  of  subject  analyses  of  five  graph  types.  Data  visualization  and 
analysis  within  DaViTo  was  a  large  part  of  this  thesis,  and  many  improvements  were 
suggested  that  would  increase  the  usability  and  assist  developers  in  meeting  the  initial 
goals  outlined  for  the  software.  The  next  subset  focused  on  three  common  questions 
across  each  session  as  well  as  an  extensive  final  survey.  With  the  wealth  of  information 
gathered,  multiple  new  and  unique  recommendations  were  formulated  to  improve  upon 
all  attributes  of  usability.  The  final  subset  of  recommendations  were  developed  by  the 
thesis  authors  and  are  a  result  of  the  hours  spent  designing  and  carrying  out  the 
experiment  sessions  and  the  accompanying  research.  They  consist  of  four  final  big 
picture  recommendations  to  assist  developers  with  the  next  revision  of  DaViTo. 

In  the  end,  37  recommendations  have  been  formulated  for  dissemination  to  TRAC 
Monterey.  Several  of  these  are  repeats;  however,  they  have  a  different  usability  attribute 
and  basis  for  recommendation.  The  full  list  of  recommendations  can  be  found  in 
Appendix  A.  Ideally,  the  fruits  of  this  thesis  will  make  it  into  future  revisions  of  DaViTo 
and  succeed  in  making  it  a  premier  software  package  useful  throughout  the  military  and 
beyond. 
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B. 


FUTURE  WORK 


Prospective  topics  for  additional  research  derived  from  this  thesis  are  abundant. 
Usability  testing  and  eye-tracking  both  have  far-reaching  capabilities  in  the  future.  The 
first  is  a  follow-up  usability  test  of  the  next  revision  of  the  DaViTo  software. 
Verification  of  improvements  in  all  attributes  of  usability  would  be  useful  to  verily  the 
recommendations  resulting  from  this  thesis  were  indeed  applied  and  beneficial.  Further 
research  is  also  necessary  in  the  field  of  data  visualization.  This  thesis  was  very  limited 
in  scope  with  regards  to  the  techniques  tested.  As  data  visualization  methods  advance 
there  is  a  need  for  research  in  this  field  to  continue  evolving  DaViTo.  Additionally,  the 
software  could  be  further  improved  and  refined  through  additional  testing.  Another  is 
research  into  the  application  of  eye-tracking  to  usability  testing.  As  a  relatively  new 
source  of  data  for  research,  eye-tracking  has  boundless  potential.  Determining  the  most 
effective  ways  to  utilize  the  process  and  resulting  data  to  maximize  improvements  to  the 
research  subject  would  be  beneficial.  Finally,  usability  testing  of  other  products  with  the 
same  methods  used  in  this  thesis  would  be  useful.  Confirmation  of  the  methods  used  to 
conduct  testing  on  DaViTo  could  lead  to  advancement  in  the  field  of  usability  testing  and 
enhance  future  product  analyses. 
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APPENDIX  A.  CONSOLIDATED  RECOMMENDATIONS 


Appendix  A  presents  a  consolidated  list  of  recommendations  for  the  DaViTo 
software  in  Table  42.  The  designator  represents  which  portion  of  results  it  came  from. 
Recommendations  based  on  task  results  have  the  designator  A,  those  from  graph  results 
are  designated  B,  the  C  designation  is  based  on  survey  results,  and  finally  the  D 
designator  is  for  author  recommendations.  The  attributes  are  described  previously  in 
Table  1.  Repeated  recommendations  are  identified  with  an  asterisk  in  the  designator,  but 
have  a  different  attribute  and/or  basis. 


Designator 

Recommendation 

Description 

Attribute  and  Basis  for  Recommendation 

A.l 

Improve  software 
processing,  response 
time,  and  stability. 

Efficiency  -  When  drawing  user-defined 
polygons  there  was  excessive  lag  time  or  the 
software  crashed. 

A.2 

Implement  a  progress 
status  bar. 

Effectiveness  -  When  drawing  polygons,  the  lag 
time  led  subjects  to  continue  clicking  outside  the 
polygon  because  there  was  no  indication  of 
progress. 

A.3 

Draw  and  populate  a 
polygon  when  drawn 
by  the  user. 

Effectiveness  -  Right  clicking  outside  of  the 
polygon  was  counter-intuitive  to  users. 

A.4 

Implement  file 
operations  in  the  main 
menu  ribbon. 

Effectiveness  -  Heat  mapping  shows  that  many 
users  looked  to  the  menu  ribbon  for  file 
operations. 

B.l 

Make  the  Stacked  Bar 
Format  (SBF)  the 
default  graph  type. 

Efficiency  -  The  SBF  graph  was  the  most 
conducive  to  generating  a  correct  answer  from 
experiment  subjects. 

B.2 

Add  more  options  for 
graph  types. 

Usefulness  -  Different  graphs  have  better 
performance  relative  to  the  question  asked. 
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Designator 

Recommendation 

Description 

Attribute  and  Basis  for  Recommendation 

C.l 

Develop  a  tutorial 
based  training  product 
for  DaViTo. 

Leamability  -  Currently  there  are  no  training 
products  for  DaViTo  beyond  a  basic  PowerPoint 
slide  show. 

C.2 

Distribute  a  training 
product  with  DaViTo. 

Leamability  -  There  is  only  a  PowerPoint  slide 
show  distributed  with  DaViTo  currently. 

C.3 

Link  the  user’s  manual 
to  the  Help  tab  on  the 
ribbon  bar. 

Usefulness  -  Currently  the  user’s  manual  is  not 
accessible  via  the  DaViTo  interface. 

C.4 

Include  graph  tools  to 
improve  graph 
readability. 

Usefulness  -  Graph  interpretation  was  identified 
by  subjects  as  the  one  of  the  most  difficult  tasks. 

C.5 

Make  graphs  easier  to 

create. 

Usefulness  -  Graph  creation  was  identified  by 
subjects  as  the  one  of  the  most  difficult  tasks. 

C.6* 

Draw  and  populate  a 
polygon  when  drawn 
by  the  user. 

Usefulness  -  Polygon  functions  were  identified 
by  subjects  as  one  of  the  most  difficult  tasks. 

C.l 

Improve  the  software 
reaction  time. 

Effectiveness  -  83  percent  of  subjects  were  not 
satisfied  with  the  software  reaction  time. 

C.8* 

Improve  software 
processing,  response 
time,  and  stability. 

Effectiveness  -  Subjects  stated  this  as  the  worst 
deficiency  that  reduced  the  simplicity  of  use  of 
DaViTo. 

C.9* 

Implement  a  progress 
status  bar. 

Effectiveness  -  Subjects  stated  this  as  the  second 
worst  deficiency  that  reduced  the  simplicity  of 
use  of  DaViTo. 

C.10 

Include  geographic 
labels  in  the  Map  and 
Data  Display  Area. 

Effectiveness  -  Subjects  stated  this  as  the  third 
worst  deficiency  that  reduced  the  simplicity  of 
use  of  DaViTo. 

C.ll* 

Improve  software 
processing,  response 
time,  and  stability. 

Efficiency  -  67  percent  of  subjects  disagreed  that 
the  software  was  stable  and  not  prone  to  crashing. 
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Designator 

Recommendation 

Description 

Attribute  and  Basis  for  Recommendation 

C.12* 

Include  geographic 
labels  in  the  Map  and 
Data  Display  Area. 

Efficiency  -  Subjects  reported  this  as  the  top 
deficiency  that  made  visual  representations  easy 
to  read. 

C.13 

Make  data  types 
within  a  data  set 
sortable  by  color. 

Efficiency  -  Subjects  reported  this  as  a  deficiency 
that  made  visual  representations  difficult  to  read. 

C.14* 

Develop  a  tutorial 
based  training  product 
for  DaViTo. 

Learnability  -  75  percent  of  subjects  reported  that 
most  people  could  figure  out  how  to  use  DaViTo 
with  the  tools  provided;  however  58  percent 
believe  more  training  products  are  needed. 

C.15* 

Distribute  a  training 
product  with  DaViTo. 

Learnability  -  88  percent  of  subjects  are  either 
neutral  or  believe  that  more  training  is  necessary 
to  effectively  be  able  to  use  DaViTo. 

C.16* 

Link  the  user’s  manual 
to  the  Help  tab  on  the 
ribbon  bar. 

Satisfaction  -  46  percent  of  users  think  they  can 
conduct  basic  tasks,  and  25  percent  of  users  think 
they  can  complete  complex  tasks,  without  the 
user’s  manual. 

C.17 

Develop  a 
comprehensive  help 
capability  for  DaViTo. 

Satisfaction  -  Only  21  percent  of  subjects  are 
comfortable  using  DaViTo  without  assistance. 
Additionally  this  was  the  second  most  frequent 
complaint  with  regards  to  satisfaction. 

C.18* 

Improve  software 
processing,  response 
time,  and  stability. 

Satisfaction  -  This  was  the  top  complaint  with 
regards  to  satisfaction. 

C.19* 

Include  graph  tools  to 
improve  graph 
readability. 

Satisfaction  -  This  was  the  third  most  frequent 
complaint  with  regards  to  satisfaction. 

C.20* 

Include  graph  tools  to 
improve  graph 
readability. 

Visualization  -  Only  12.5  percent  of  subjects 
agreed  that  output  data  is  clear  to  interpret,  and 
the  second  most  frequent  response  to  desired 
changes  to  the  output  was  more  graph 
manipulation  tools. 
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Designator 

Recommendation 

Description 

Attribute  and  Basis  for  Recommendation 

C.21* 

Add  more  options  for 
graph  types. 

Visualization  -  The  most  frequent  response  to 
desired  changes  to  software  output  was  more 
graph  options. 

C.22* 

Make  data  types 
within  a  data  set 
sortable  by  color. 

Visualization  -  Only  12.5  percent  of  subjects 
agreed  that  displayed  output  data  is  clear  to 
interpret,  and  the  third  most  frequent  response  to 
desired  changes  to  the  output  was  a  greater 
capability  to  tailor  the  data. 

C.23 

Alter  the  cursor 
navigation  button 

+1+ 

to  match  the  map 
navigation  icon. 

Rta 

m 

Visualization  -  54  percent  of  subjects  could  not 
correctly  identify  the  cursor  navigation  icon.  7 1 
percent  of  subjects  could  correctly  identify  the 
map  navigation  icon.  Some  incorrect  answers  for 
each  were  the  other  icons  title.  The  functionality 
associated  with  these  icons  is  very  similar. 

C.24 

Standardize  two 

button  mouse 
functionality. 

Multiple  Attributes  -  Follow  the  standard 
functionality  for  a  two-button  mouse  that  most 
software  developers  utilize. 

C.25 

Incorporate  an  area 
relief  option  or  3D 
capability  similar  to 
Google  Earth. 

Visualization  -  Top  response  when  subjects  were 
asked  what  they  would  change  about  the  color 
scheme  or  user  interface  layout. 

C.26 

The  color  scheme  is 

bland.  Add  more 

contrast. 

Visualization  -  In  an  effort  to  look  more  inviting 
to  users  software  should  have  visual  appeal,  even 
in  military  applications. 

C.21 

Change  the  default 
data  point  color. 

Visualization  -  The  current  default  of  gray  has  no 
contrast  with  the  interface  due  to  frequent  use  of 
different  shades  of  gray. 
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Designator 

Recommendation 

Description 

Attribute  and  Basis  for  Recommendation 

D.l 

Add  a  collapse 
capability  to  the  Data 
Sets,  Predefined 
Polygons,  and  Regions 
Selection  List  areas  of 

the  interface 

Visualization  -  This  will  expose  more  of  the  Map 
and  Data  Display  area,  reducing  clutter  and 
allowing  the  user  more  capability  to  work  within 
the  primary  workspace  of  the  interface. 

D.2 

Develop  training 
products  for  both  a 
basic  user  and 
experienced  analyst. 

Usefulness  -  Recommendations  C.l  and  C.2 
address  the  development  of  training  products. 
There  are  complex  statistical  capabilities  in 
DaViTo.  An  analyst  with  advanced  skills  would 
benefit  from  tailored  training  that  goes  beyond  the 
capabilities  in  the  user  interface. 

D.3 

Extend  mapping 
capability  to  include 
nautical  charts. 

Usefulness  -  Many  Navy  subjects  identified 
additional  uses  such  as  counter-piracy  or  sea- 
based  counter-drug  operations. 

D.4 

Develop  a 

standardized  Army  (or 
service  wide)  database 
and  data  formats  for 

use  with  DaViTo. 

Usability  -  Without  a  fonnal  database  and  data 
formats  the  tool  itself  may  suffer  reduced 
usability. 

Table  42.  Consolidated  list  of  recommendations  for  the  improvement  of  DaViTo  and 
their  basis  derived  from  all  research  methods. 
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APPENDIX  B.  ADDITIONAL  TASK  RESULTS 


Appendix  B  contains  cumulative  gaze  frequency  data  for  all  final  session  tasks  as 
heat  maps  in  Figures  19  through  26.  Figures  27  through  36  show  the  cumulative  counts 
of  gaze  observations  for  all  subjects  in  each  region  of  the  interface  for  the  heat  maps. 
Heat  maps  and  gaze  observation  graphs  presented  in  the  body  of  the  thesis  are  not 
reproduced  in  this  Appendix. 
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Figure  19.  Final  session  task  two  heat  map  and  mouse  clicks.  Task  two  directed  subjects  to  load  the  Kenya  Roads  Shapefile,  which 
required  interaction  with  the  Shapefiles  region. 
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Figure  20.  Final  session  task  three  heat  map  and  mouse  clicks.  Task  three  directed  subjects  to  go  to  the  Africom  view. 
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Figure  2 1 .  Final  session  task  five  heat  map  and  mouse  clicks.  Task  live  directed  subjects  to  close  and  delete  the  chart  generated  in 
task  four. 
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Figure  22.  Final  session  task  six  heat  map  and  mouse  clicks.  Task  six  directed  subjects  to  hide  the  Map  Controls  ribbon  bar. 
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Figure  23.  Final  session  task  seven  heat  map  and  mouse  clicks.  Task  seven  directed  subjects  to  navigate  to  Iraq  without  the  use  of 
the  Map  Controls  portion  of  the  interface. 
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Figure  24.  Final  session  task  eight  heat  map  and  mouse  clicks.  Task  eight  directed  subjects  to  go  back  to  the  previous  view. 


103 


Figure  25.  Final  session  task  nine  heat  map  and  mouse  clicks.  Task  nine  directed  subjects  to  remove  all  data  sets  and  Shapefiles,  then 
load  the  IndureDemo  data  set  and  the  afghanistan_provincial_boundaries  Shapefile. 
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Figure  26. 


Final  session  task  eleven  heat  map  and  mouse  clicks.  Task  eleven  asked  subjects  to  answer  a  trend  analysis  question  using 
the  graph  generated  in  the  user  defined  polygon  from  task  ten. 
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Gaze  Observations  In  Each  Region 


Figure  27.  Graph  of  final  session,  task  two,  cumulative  counts  of  gaze  observations  for 
all  subjects.  Task  two  directed  subjects  to  load  the  Kenya  Roads  Shapefde, 
which  required  interaction  with  the  Shapefiles  region.  Counts  correlate  to  the 
number  of  subject  fixations  in  each  region  shown  in  Figure  19. 


Gaze  Observations  In  Each  Region 


Figure  28.  Graph  of  final  session,  task  three,  cumulative  counts  of  gaze  observations  for 

all  subjects.  Task  three  directed  subjects  to  go  to  the  Africom  view.  Counts 
correlate  to  the  number  of  subject  fixations  in  each  region  shown  in  Figure 
20. 
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Gaze  Observation*  in  Each  Region 


Figure  29.  Graph  of  final  session,  task  four,  cumulative  counts  of  gaze  observations  for 
all  subjects.  Task  four  directed  subjects  to  create  a  user  defined  polygon  and 
display  the  chart.  Counts  correlate  to  the  number  of  subject  fixations  in  each 
region  shown  in  Figure  9. 


Gaze  Observations  in  Each  Region 


Figure  30.  Graph  of  final  session,  task  five,  cumulative  counts  of  gaze  observations  for 
all  subjects.  Task  five  directed  subjects  to  close  and  delete  the  chart 
generated  in  task  four.  Counts  correlate  to  the  number  of  subject  fixations  in 
each  region  shown  in  Figure  2 1 . 
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Gaze  Observations  in  Each  Region 


Figure  3 1 .  Graph  of  final  session,  task  six,  cumulative  counts  of  gaze  observations  for 

all  subjects.  Task  six  directed  subjects  to  hide  the  Map  Controls  ribbon  bar. 
Counts  correlate  to  the  number  of  subject  fixations  in  each  region  shown  in 
Figure  22. 


Gaze  Observations  in  Each  Region 


Figure  32.  Graph  of  final  session,  task  seven,  cumulative  counts  of  gaze  observations  for 
all  subjects.  Task  seven  directed  subjects  to  navigate  to  Iraq  without  the  use 
of  the  Map  Controls  portion  of  the  interface.  Counts  correlate  to  the  number 
of  subject  fixations  in  each  region  shown  in  Figure  23. 
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Gaze  Observations  In  Each  Region 


Figure  33.  Graph  of  final  session,  task  eight,  cumulative  counts  of  gaze  observations  for 
all  subjects.  Task  eight  directed  subjects  to  go  back  to  the  previous  view. 
Counts  correlate  to  the  number  of  subject  fixations  in  each  region  shown  in 
Figure  24. 


Gaze  Observations  In  Each  Region 


Figure  34.  Graph  of  final  session,  task  nine,  cumulative  counts  of  gaze  observations  for 
all  subjects.  Task  nine  directed  subjects  to  remove  all  data  sets  and 
Shapefiles,  then  load  the  IndureDemo  data  set  and  the 
afghanistan_provincial_boundaries  Shapefile.  Counts  correlate  to  the 
number  of  subject  fixations  in  each  region  shown  in  Figure  25. 
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Gaze  Observations  in  Each  Region 


Figure  35.  Graph  of  final  session,  task  ten,  cumulative  counts  of  gaze  observations  for 
all  subjects.  Task  ten  directed  subjects  to  answer  a  question  that  required  the 
creation  of  a  user  defined  polygon,  and  to  display  the  chart.  Counts  correlate 
to  the  number  of  subject  fixations  in  each  region  shown  in  Figure  10. 


Gaze  Observations  in  Each  Region 


Figure  36.  Graph  of  final  session,  task  eleven,  cumulative  counts  of  gaze  observations 
for  all  subjects.  Task  eleven  asked  subjects  to  answer  a  trend  analysis 
question  using  the  graph  generated  in  the  user  defined  polygon  from  task  ten. 
Counts  correlate  to  the  number  of  subject  fixations  in  each  region  shown  in 
Figure  26. 
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APPENDIX  C.  ADDITIONAL  GRAPH  RESULTS 


Appendix  C  contains  additional  results  from  the  analysis  of  the  data 
representation  portion  of  the  experiment.  Table  43  presents  the  data  representation 
questions  asked  about  each  graph  type.  The  statistical  analyses  of  the  questions  in  Table 
43  are  presented  in  Tables  44  through  50  and  Figures  37  through  43. 


Question 

Designation 

Session 

Question  Text 

A 

One 

What  were  the  total  events  for  December  2012? 

B 

One 

What  event  had  the  highest  frequency  in  Jun  2013? 

C 

One 

What  month  had  the  most  events? 

D 

Two 

What  month  had  the  least  events? 

E 

Two 

Which  month  had  the  highest  frequency  for  event  D? 

F 

Three 

Compare  the  months  of  December  2012  and  December 

2013.  What  was  the  increase  or  decrease  in  total  events? 

G 

Three 

What  event  had  the  highest  sum  for  the  months  of  February 
2012  and  February  2013? 

Table  43.  List  of  questions  for  each  session  and  their  designation  letter.  This  is  a  repeat 
of  Table  12  in  Chapter  IV,  Section  B,  shown  here  to  correlate  data  to 
questions. 


Question 

Accuracy 

Time 

A 

(%  Correct) 

(sec) 

CCF 

41.67 

82.30 

SBF 

70.83 

34.90 

LCF 

33.33 

40.59 

SLMF 

8.33 

46.19 

SAF 

33.33 

35.56 

Table  44.  Statistics  for  session  one,  question  A,  for  all  graph  types. 
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Figure  37.  Plot  of  session  one,  depicting  time  to  answer  versus  accuracy  of  the  answer 
for  question  A.  Note  the  decreasing  time  on  the  Y-axis.  Top  performance  is 
in  upper  right  corner. 


Question 

Accuracy 

Time 

B 

(%  Correct) 

(sec) 

CCF 

75.00 

26.91 

SBF 

79.17 

25.77 

LCF 

58.33 

23.69 

SLMF 

12.50 

21.34 

SAF 

41.67 

16.29 

Table  45.  Statistics  for  session  one,  question  B,  for  all  graph  types. 


Session  1,  Question  B  Time  vs  Accuracy 

Accuracy  (%  correct) 
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Figure  38.  Plot  of  session  one,  depicting  time  to  answer  versus  accuracy  of  the  answer 
for  question  B.  Note  the  decreasing  time  on  the  Y-axis.  Top  performance  is 
in  upper  right  corner. 
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Table  46. 


Figure  39. 


Table  47. 


Question 

Accuracy 

Time 

C 

(%  correct) 

(sec) 

CCF 

45.83 

65.78 

SBF 

91.67 

22.77 

LCF 

50.00 

50.85 

SLMF 

45.83 

33.00 

SAF 

79.17 

22.70 

Statistics  for  session  one,  question  B,  for  all  graph  types. 


Session  1,  Question  C  Time  vs  Accuracy 
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Plot  of  session  one,  depicting  time  to  answer  versus  accuracy  of  the  answer 
for  question  C.  Note  the  decreasing  time  on  the  Y-axis.  Top  performance  is 
in  upper  right  corner. 


Question 

Accuracy 

Time 

D 

(%  Correct) 

(sec) 

SAF 

83.33 

28.27 

SLMF 

87.50 

24.30 

LCF 

25.00 

61.37 

SBF 

87.50 

20.85 

CCF 

79.17 

40.00 

Statistics  for  session  two,  question  D,  for  all  graph  types. 
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Figure  40. 


Session  2,  Question  D  Time  vs  Accuracy 
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Plot  of  session  two,  depicting  time  to  answer  versus  accuracy  of  the  answer 
for  question  D.  Note  the  decreasing  time  on  the  Y-axis.  Top  performance  is 
in  upper  right  corner. 


Question 

Accuracy 

Time 

E 

(%  Correct) 

(sec) 

SAF 

66.67 

40.97 

SLMF 

58.33 

34.32 

LCF 

75.00 

24.21 

SBF 

41.67 

36.92 

CCF 

87.50 

20.77 

Table  48.  Statistics  for  session  two,  question  E,  for  all  graph  types. 


Session  2,  Question  E  Time  vs  Accuracy 
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Figure  41 .  Plot  of  session  two,  depicting  time  to  answer  versus  accuracy  of  the  answer 
for  question  E.  Note  the  decreasing  time  on  the  Y-axis.  Top  performance  is 
in  upper  right  corner. 
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Question 

Accuracy 

Time 

F 

(%  Correct) 

(sec) 

LCF 

12.50 

107.13 

SLMF 

33.33 

67.39 

SAF 

50.00 

49.07 

SBF 

75.00 

44.33 

CCF 

50.00 

59.10 

Table  49.  Statistics  for  session  three,  question  F,  for  all  graph  types. 


Session  3,  Question  F  Time  vs  Accuracy 

Accuracy  (%  correct) 

0  10  20  30  40  50  60  70  80  90  100 

40 

50 

_  60 

A  70 

g  80 

p  90 
100 

110 

120 

♦  CCF 

■  S3F 

A  LCF 

XSLMF 

)tC  SAF 

X  SAF  SBF  " 

SLMF 

A  1  CF 

Figure  42.  Plot  of  session  three,  depicting  time  to  answer  versus  accuracy  of  the  answer 
for  question  F.  Note  the  decreasing  time  on  the  Y-axis.  Top  performance  is 
in  upper  right  corner. 


Question 

Accuracy 

Time 

G 

(%  Correct) 

(sec) 

LCF 

87.50 

67.50 

SLMF 

62.50 

52.93 

SAF 

70.83 

42.31 

SBF 

75.00 

33.86 

CCF 

100.00 

50.15 

Table  50.  Statistics  for  session  three,  question  G,  for  all  graph  types. 
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Session  3,  Question  G  Time  vs  Accuracy 

Accuracy  [%  correct) 
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Figure  43.  Plot  of  session  three,  depicting  time  to  answer  versus  accuracy  of  the  answer 
for  question  G.  Note  the  decreasing  time  on  the  Y-axis.  Top  performance  is 
in  upper  right  corner 
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APPENDIX  D.  ADDITIONAL  SURVEY  DATA 


A.  THE  THREE  COMMON  QUESTIONS 


Appendix  D,  part  A,  presents  additional  results  from  the  three  questions  common 
to  each  session.  The  results,  shown  in  Figures  44  through  48,  are  for  each  individual 
question.  Note  that  the  answers  to  the  free-fonn  questions  in  Tables  51  through  53  are 
not  altered  in  content  from  what  the  subjects  entered  in  the  survey. 


Session  1  Question  1:  How  would  you 
rate  your  first  training  session? 


Figure  44.  Session  one,  question  one,  with  responses  on  a  scale  of  one  (very  poor)  to 
five  (very  good.) 


Figure  45. 
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Session  1  Question  2:  How  many 
more  training  hours  are  needed? 
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■  Response  4 
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Session  one,  question  two,  subject’s  self-evaluation  of  additional  training 
hours  needed. 
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Session  One,  Question  Three:  What  were  your  top  three  likes  and  dislikes  with 

your  first  exposure  to  DaViTo? 

Likes:  Visual  display  of  data  possible  for  "walk  through"  with  a  boss,  it  seems  like 
you  could  create  and  share  databases  with  others  to  share  the  same  view  of  a  situation, 
shapes/displays/colors  are  customizable. 

Dislikes:  It  might  be  chugging  through  a  lot  of  data  when  you  move  the  map  each 
time:  seems  to  go  more  slowly  than  necessary.  Otherwise,  need  more  exposure  to 

make  a  better  judgment. _ 

Likes:  1)  Easy  to  use  2)  The  tab  identifies  are  descriptive  3)  Quick  to  learn 
Dislikes:  1)  Couldn't  remember  the  zoom  in  2)  The  distance  circle  is  deceiving  3) 

Map  changes  slowly _ 

Likes:  1.  Easy  to  navigate  2.  Overlays  3.  Quick  reference 

Dislikes:  1.  Seems  like  it  runs  (scrolls)  slowly  2.  Unable  to  directly  click  on  data 

points  when  using  cursor3.  No  city  or  population  center  overlays. _ 

Likes:  All  of  the  data  is  displayed  in  one  window  making  it  easy  to  use.  Good 
introductory  tutorial.  Tool  for  zooming  in  on  certain  areas  of  interest  is  useful 
Dislikes:  Lag  time  in  updating  the  map.  Map  controls  would  be  more  useful  if  on  the 

map.  Zoom  and  pan  functions  could  use  more  fine  controls. _ 

Likes:  Good  Company-  Fun  presentation-  Cool  gadgets. 

Dislikes:  Slow  gui-  no  map  labels-  distance  tool  should  have  separate  way  points _ 

Likes:  Simple  navigation  toolbar-  saved  views- 

Dislikes:  Measurement  tool  graphics  leave  an  artifact  sometimes-  slow  refresh  rate- 
Likes:  1.  Easy  to  use  2.  Easy  to  understand  the  interface  3.  Redundant  ways  to 
accomplish  the  same  task 

Dislikes:  1.  No  preloaded  options  for  US  layers  (geographical  boundaries)  2.  Not  clear 

on  how  to  add/create  additional  layers  3.  No  built-in  tutorials _ 

Likes:  1  -  common  menus  and  icons  to  other  similar  software  -  easier  to  leam  2  - 
simple-  plain  interface.  3  -  easy  to  get  to  the  right  functions. 

Dislikes:  1  -  speed  of  map  refresh  -  likely  a  function  of  the  hardware-  but  it  was  a  little 
annoying  to  see  the  lag  between  a  selection  that  the  new  map  2  -  "MGRS"  was 

misspelled  in  the  dropdown  menu  -  was  written  "MRGS" _ 

Likes:  1 .  My  first  like  is  the  usability  of  the  navigation  functions.  They  are  very  easy 
to  understand  and  apply.  2.  My  second  like  is  the  use  of  a  importation  of  data  sets 
from  pre-existing  files.  3.  My  third  like  is  the  ease  of  understandability  of  how  to  use 
the  program. 

Dislikes:  1.  My  first  dislike  is  the  time  lag  in  the  zoom  in-  zoom  out  function. 
2.  My  second  dislike  is  the  caliper  function  of  the  use  of  distance.  This  tool  needs  to  be 
updated  to  something  that  is  more  user  friendly  and  easily  shows  data  that 

you  are  trying  to  find  to  new  users. _ 
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Session  One,  Question  Three:  What  were  your  top  three  likes  and  dislikes  with 

your  first  exposure  to  DaViTo? 

Likes:  1.  Liked  that  it  was  pretty  simple  to  use.  2.  There  wasn't  too  much  information 
(overload)  on  the  screen.  3.  It  also  seems  familiar  with  other  applications  (like  GPS)  so 
that  made  it  easy  to  figure  out. 

Dislikes:  1.1  didn’t  like  that  it  seemed  slow  to  respond.  I  feel  like  I  could  easily  over 
task  the  computer  if  I  was  trying  to  get  infonnation  quickly.  2.  It  took  me  a  second 
(with  help)  to  figure  out  how  to  use  the  measurement  tool.  3.  While  I  liked  that  there 

wasn’t  information  overload-  at  the  same  time  I  felt  it  was  too  generic  and  simple. _ 

Likes:  I  liked  how  the  tools  were  easy  to  see  and  use  on  the  left  panel.  I've  have 
experience  with  other  military  mapping  programs  and  they  have  always  confused  me 
on  how  mapping  data  is  loaded  and  accessible  by  the  user. 

Dislikes:  I  don't  like  how  slow  the  refresh  rate  it.  I  would  get  frustrated  using  this 
program  in  a  stressful  environment  when  I  would  start  clicking  to  many  times  and  get 

into  a  "do-loop”  because  the  program  was  too  slow.  That  is  about  it. _ 

Likes:  Multiple  options  to  manipulate  -  redundancy-  user  interface  is  intuitive-  color 
are  not  distracting  and  appropriate 

Dislikes:  System  latency-  zoom  option  (the  squared  icon  on  the  tool  bar)  when 
selecting  a  location  hard  to  center-  no  country  labels  provided  (similarly-  states- 

provinces-  districts-  etc.) _ 

Likes:  Easy  to  see  the  maps  on  the  right  display.  Once  I  learned  the  navigation 
portion-  it  was  easy  to  zoom  in  on  a  specific  area.  Easy  to  load  data  sets. 

Dislikes:  I  got  confused  on  zooming  in  with  the  mouse  scroll;  I  went  the  opposite 

direction. _ 

Likes:  1)  Fairly  intuitive  2)  Efficient  selection  of  functionality  (no  more  a  couple 
clicks  to  do  what  I  wanted)  3)  Simplicity  (functionality  I  was  instructed  in  during  this 
session  was  all  visible  on  the  screen) 

Dislikes:  1)  Didn’t  like  the  position  of  lat/long  and  distance  down  in  the  left  corner 
(my  preference  would  be  maybe  in  the  top  middle-  closer  to  where  I  was  looking 
(ideally  the  distance  would  show  up  in  the  middle  of  the  great  circle  distance  as  you 
measure)  2)  Zoom  box  function  was  refined  (it  worked  but  didn’t  allow  me  to  drag 
down  to  the  lower  right  like  I  intended)  3)  Appears  to  have  limited  level  functionality 

compared  to  something  like  CPOF  (where  you  could  analyze  in  3D) _ 

Likes:  Easy  to  understand  icons  (intuitive  graphical  representation  +  mouse  over 
tooltip)  2:  Clean  interface-  no  clutter  3:  Controls  are  similar  to  nonnal  software  (e.g. 
pan  using  click  and  drag-  zoom  using  selection  box  or  mouse  wheel) 

Dislikes:  1:  Response  Lag-  especially  when  click-and-drag  to  pan  when  data  sets  are 
loaded  2:  Text  is  a  bit  small  (necessary  tradeoff  to  maximize  map  display  area?)  3. 
Info  bar  (where  it  shows  the  Lat/Long/Distance)  at  bottom  left  is  small  and  tucked 
away.  Hard  to  use  with  the  distance  measurement  function. _ 
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Session  One,  Question  Three:  What  were  your  top  three  likes  and  dislikes  with 

your  first  exposure  to  DaViTo? 

Dislike:  Information  could  be  displayed  in  conjunction  with  a  map  built  in.  Did  not 
like  the  system  being  "jittery"  when  navigating  on  the  map.  Could  lose  the  location 
you  are  looking  for  when  moving  around  do  to  the  slow  response  from  mouse  input. 
Needs  to  be  smoother.  Slow,  No  country  labels,  Distance  ring  to  thin,  Distance  tools 
doesn't  allow  you  to  make  separate  waypoints 

Likes:  Easy  to  use. _ 

Likes:  1)  Multiple  map  projections  available.  2)  Clearly  defined  data  set  types  along 

with  data  set  previews.  3)  Ease  of  data  set  integration  into  map. _ 

Likes:  1.  Ease  of  use.  Resembles  other  map  programs.  2.  Ability  to  quickly  toggle 
between  chosen  projections  without  using  the  zoom  feature.  3.  Theater  drop  down  to 
focus  on  the  different  AORs. 

Dislikes:  1.  Distance  tool  line  blends  in  with  rest  of  map  lines.  Not  distinguished. 
Likes:  Ease  of  use,  Convenient  lay  out. 

Dislikes:  Lack  of  labels _ 

Likes  -  1. Simple  menus-  2.  Quick  feedback  when  a  tool  was  used  (distance-  zoom) 

Dislikes  -  1 .  not  sure  yet _ 

Likes:  user  friendly  dislikes:  when  using  the  pan  tool  the  delay  caused  by  moving  the 

map _ 

Likes:  1)  Very  user  friendly  2)  Good  graphical  user  interface  3)  Follows  standard 
format  like  most  Windows  programs-  so  if  you  are  a  Windows  user  it  was  easy  to 

understand.  Dislikes:  1)  A  little  slow  with  zooming  out _ 

Likes:  Simplicity-  Familiarity-  distance  measurement 

Dislike:  Scaling  back  out-  no  scroll  wheel  use-  having  to  come  up  with  three  dislikes 
Likes:  Ease  of  use.  Ease  of  understanding  of  the  functionality  of  the  tool.  Simplicity 
of  functions 

Dislikes:  Latency  of  program  reaction  to  user  demand  D:  Layout  of  Screen  D:  Default 
color  scheme;  should  have  used  Army  colors _ 

Table  51.  Table  of  likes  and  dislikes  for  all  24  subjects  in  session  one.  Comments 
taken  directly  from  the  EyeWorks  software  with  no  modification. 
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Session  2  Question  1:  How  would  you 
rate  your  second  training  session? 


Figure  46.  Session  two,  question  one,  with  responses  on  a  scale  of  one  (very  poor)  to 
five  (very  good.) 


Figure  47. 


Session  2  Question  2:  How  many 
more  training  hours  are  needed? 
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Session  two,  question  two,  subject’s  self-evaluation  of  additional  training 
hours  needed. 
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Session  Two,  Question  Three:  What  were  your  likes  and  dislikes  with  your 

second  exposure  to  DaViTo? 

Likes:  Overall  like  the  product  and  potential. 

Dislikes:  The  functionality  associated  with  the  shapes  gave  me  a  bit  of  trouble 
(following  the  tutorial  and  associated  "outside  the  circle  left-click"  didn't  work  at  first. 

Likes:  Easy  to  understand.  Helps  to  have  the  tutorial  run  through  the  steps.  Would 
take  more  time  using  it  to  get  all  the  steps  correct.  Just  more  practice  time  on  the  tasks 

that  need  to  be  accomplished. _ 

Dislikes:  I  did  not  like  that  the  graph  display  did  not  have  an  easy  method  of  changing 
the  axis  increments.  The  recall  of  graphs  also  seemed  to  take  a  little  effort. 

Likes:  The  display  of  the  charts  for  the  different  regions  is  helpful. 

Dislikes:  The  right  click  function  did  not  work  to  display  the  full  menu  on  several 
occasions-  on  the  map  and  the  chart  view.  It  would  be  useful  to  have  an  hourglass  to 
show  progress  when  loading  data  sets. 

Likes:  I  enjoyed  learning  more  of  the  capabilities  of  DaViTo 

Dislikes  The  right-left  mouse  use  when  creating  and  exploring  charts 

Likes:  Liked  the  way  you  could  get  to  a  graph  so  quickly  based  on  just  drawing  a 

Shapefile  on  the  chart.  Very  convenient. 

Disliked:  The  way  you  had  to  repeatedly  right  click  on  the  smaller  graphs  to  bring  up 
the  options  menu.  Initially-  all  that  is  available  is  the  create  option  (first  four  to  five 

clicks). _ 

Likes:  Easy  to  use  2.  Shape  tools  to  select  particular  regions  were  intuitive.  Chart 
output  was  easy  to  access  and  read 

Dislikes:  1.  No  option  for  irregular  shapes  2.  Data  analysis  tools  not  explained  in- 

depth _ 

Likes:  Seems  like  an  easy  way  of  embedding  data  into  a  map  to  make  it  more 
meaningful. 

Dislikes:  The  lag  between  mouse  clicks  and  the  menu  appearing  is  frustrating-  but  not 
as  frustrating  as  the  Right  click  menus  not  acting  the  same  each  time.  Sometimes  I 
would  get  a  short  list  of  options-  other  times  a  longer  list  of  options.  This  should  be 
more  consistent. 

Likes:  I  do  like  the  capability  of  showing  terrain  on  the  graph  and  also  the  function  of 
showing  data  per  data  "dot". 

Dislike:  Also-  loading  time  is  still  slow.  I  think  that  exposure  for  more  than  15 
minutes  is  required  to  gain  a  full  grasp. 

Dislikes:  It  was  quite  a  bit  harder  for  me  to  navigate  through  the  charts-  but  with 
repetition  I  should  be  fine.  So  I  disliked  having  to  play  "seek  and  find"  to  figure  out 

what  I  needed.  I  really  don't  have  any  likes  at  this  time. _ 

Likes:  liked  the  way  you  can  bring  up  data  on  specific  events. 

Dislikes:  I  didn’t  like  the  slowness  of  the  program.  There  appears  to  be  decent  amount 
of  lag.  Overall-  it  seems  cluttered-  lots  of  stuff  to  see  and  not  very  well  organized.  I 
don’t  see  how  anyone  can  get  anything  out  of  the  thumb  nails  of  the  graphs  that  appear 
on  the  map.  Plus  they  occlude  the  information  under  them. 
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Session  Two,  Question  Three:  What  were  your  likes  and  dislikes  with  your 

second  exposure  to  DaViTo? 

Like:  Intuitive  to  use  select  a  specific  area  and  use  shapes  to  draw  graphs.  The 
different  options  to  modify  plots  (e.g.  -  freq  vs  percent-  normalized-  etc.) 

Dislike:  Since  there  is  an  inherent  lag  in  the  system-  a  user  feedback  prompt  or  in¬ 
progress  bar  would  be  helpful  to  the  user.  Inability  to  change  the  line  colors  in  the 

plots _ 

Likes:  Reviewed  previous  Session-  so  that  quickly  got  me  reoriented  to  the  tool. 
Gained  knowledge  on  the  system  to  include  informational/statistical  plots.  Focused 

was  on  the  different  views  and  map  manipulation. _ 

Likes:  Clear  instructions-  good  examples  of  tasks.  Hands  on  is  definitely  a  benefit  to 
learning  the  application  rather  than  just  watching  the  video. 

Dislike:  Is  really  my  personal  preference  to  see  the  instructions  while  I'm  executing 
the  task.  Since  I  don't  consider  myself  to  be  proficient  with  this  software  yet-  I’d  like 
the  option  to  refer  to  the  question  you're  asking  as  I  go  down  the  menus  (to  confirm  I 
understand  the  task). 

Like:  Step  by  step  instructions 

Dislike:  Need  to  memorize  each  step  of  task  (the  white  screen)  and  no  able  to  go  back 

to  reread  the  instruction. _ 

Dislikes:  Still  a  little  difficult  to  remember  how  to  load  data  set.  May  have  been  to  the 
brief  exposure. 

Likes:  Ease  of  general  statistical  analysis-  access  to  overlays-  ability  to  create  new 
graphs  on  the  fly. 

Dislikes:  Response  times  to  clicks-  load  times  for  data  sets-  unclear  chart  option 
explanations  in  control  panel-  too  many  steps  to  draw  a  shape. 

Like:  The  ability  to  create  the  chart  in  order  to  pull  data. 

Dislike:  Remembering  the  steps  to  create  the  chart. 

Dislikes:  The  maps  take  a  while  to  upload-  seems  okay  for  this  environment  but  it  may 
be  too  restrictive  in  a  real  work  environment  that  requires  more  changing-  uploading 

Dislikes:  Maps  and  charts  loaded  a  little  slow-  but  other  than  that  no  issues. _ 

Likes:  Good  graphical  user  interface-  and  the  pop-up  menus  were  easy  to  follow. 
Graph  displays  were  good.  No  dislikes  with  my  second  exposure. 

Likes:  Visual  interface 
Dislike:  Area  selection  method 

Dislikes:  Deeper  you  go  into  the  functionality  the  more  you  have  to  remember.  Tough 
to  do  when  you  get  older  and  your  short  term  memory  begins  to  slip. 

Table  52.  Table  of  likes  and  dislikes  for  all  24  subjects  in  session  two.  Comments 
taken  directly  from  the  EyeWorks  software  with  no  modification. 


123 


Figure  48. 


Session  3  Question  2:  How  many 
more  training  hours  are  needed? 


Session  three,  question  two,  subject’s  self-evaluation  of  additional  training 
hours  needed. 


Session  Three,  Question  Three:  What  were  your  likes  and  dislikes  with  your  third 

exposure  to  DaViTo? 

Likes:  The  interface  works  well  for  navigation  and  selection 

Dislikes  But  some  of  the  analysis  through  the  one  chart  I  know  how  to  make  was 
difficult.  If  there  is  an  icon  that  indicates  the  system  is  processing  data-  I  don't  know 
where  it  is  and  so  would  continue  to  click  on  different  icons  with  no  response. _ 

Dislikes:  Just  don't  remember  enough  of  the  tasks.  It  is  laid  out  in  a  familiar  fashion.  But 
some  of  the  later  tasks- 1  just  couldn't  remember  how  to  do. 

Dislikes:  Program  responded  sluggishly.  An  easier  way  to  access  chart  information 
would  be  beneficial.  Perhaps  a  chart  display  addition  to  the  tool  bar.  Count  functions 
would  also  be  useful  (i.e.  count  occurrences  between  this  date  and  that  date). 

Dislikes:  The  chart  for  IED  attacks  was  difficult  to  read-  it  was  hard  to  tell  which  time 
frame  corresponded  to  the  year  and  it  was  hard  to  trace  a  data  point  to  a  number  on  the  y 
axis.  There  was  no  indication  that  the  polygon  was  loaded  and  then  there  was  a  long 
delay  if  the  polygon  was  selected  before  it  was  ready. 

Likes:  Fun 

Dislikes:  Like  playing  with  the  software- 1  did  not  like  the  lack  of  progress  display. _ 

Dislikes:  Some  menu  options  are  in  counterintuitive  locations  (opening  Shapefiles  and 
map  layers).  I  couldn't  remember  how  to  create  one  of  the  graphs-  and  searching  the 
usual  areas  didn't  give  any  clues.  Once  the  graphs  are  readily  available-  the  overlay  onto 
a  geographic  area  is  convenient. _ 

Likes:  Simple  instructions  Tasks  were  well  defined 

Dislikes:  Country  not  labeled  wrt  task  given  (I  didn't  know  where  Kenya  was)  2.  No 
fantastically  narrated  training  session  by  LT  Cutler3.  Couldn't  figure  out  how  to  change 
the  display  of  chart  from  individual  event  line  to  cumulative  bar. 
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Session  Three,  Question  Three:  What  were  your  likes  and  dislikes  with  your  third 

exposure  to  DaViTo? 

Likes:  Similar  interface  as  other  C2  systems  in  the  Army  (but  difficult  to  learn  the  first 
time). 

Dislikes:  The  long  load  times  with  any  indicator  of  what  the  computer  was  doing  led  to 
some  mistakes  and  frustration.  Overall-  seemed  slow.  For  the  geographically- 
challenged-  adding  location  names  would  be  helpful.  Such  as  labeling  the  nation  names. 
Dislikes:  Slow  for  loading 

Dislikes:  It  sometimes  took  a  while  to  load  maps.  And  after  it  loaded  the  maps  I  was 
unsure  that  it  had  completed.  This  may  just  be  due  to  a  slow  processor.  Additionally-  the 
response  to  my  right  click  was  sometimes  unresponsive  or  took  too  long.  So  it  didn’t  feel 
right  and  I  would  have  to  try  another  way/method  to  accomplish  a  task.  I  also  didn’t  like 
that  when  I  was  to  load  a  shape  of  some  sort-  I  needed  to  look  for  the  polygon.  That 
didn't  make  sense  to  me  and  it  took  me  a  bit  (with  some  prompting)  to  get  where  I 
needed  to  go.  As  for  likes- 

Likes:  I  seemed  to  be  able  to  navigate  better  since  this  was  my  third  session.  But  there 

still  is  a  learning  curve. _ 

Dislikes:  SLOWMI  spend  too  much  time  "swimming".  Did  they  write  this  thing  in 
visual  basic?  Learn  to  code-  code  monkeys. 

Dislikes:  User  feedback  indicating  loading  in-progress,  a  progress  bar  would  improve 
user  interface  When  loading  a  new  data  set  or  Shapefile  the  system  does  not  pan  to  that 
location  automatically  When  using  the  right  click  in  the  map  option-  the  system  displays 
different  menu  option  some  more  complete  that  others  but  it  is  not  clear  to  the  user 
where  to  select  in  the  map  to  get  the  right  options 

Dislikes:  1.  It  takes  too  long  to  load  the  data  sets.  2.  If  the  system  is  working-  there 
needs  to  be  a  message  that  states  it  is  doing  so.  Still  easy  to  fumble  through/navigate 
+Charts  =  Awful  - 

Likes:  Once  loaded  with  data-  the  tool  provides  very  detailed  charts  for  several  types  of 

data.  Although  small-  the  charts  were  easy  to  read  based  on  color-coding. _ 

Dislikes:  Time  to  process  the  data  was  a  little  confusing  as  to  whether  I  had  messed  up 
or  the  processing  was  taking  long/computer  froze?  Country  labels  would  have  been 
helpful  with  the  boundaries  (maybe  put  some  hot  spot  labels  when  you  hover  over  each 
province/country).  When  trying  to  hide  the  map  controls  I  reverted  back  to  a  "Windows 
mentality"  looking  for  +  or  -  buttons  to  minimize-  or  hide-  the  map  controls.  Multiple 
clicks  were  probably  the  biggest  concern  I  had  during  this  session-  since  the  mouse  icon 
still  moved.  It  lead  me  to  doubt  if  I  had  selected  the  proper  command(s). 

Likes:  Interface  is  still  manageable-  despite  forgetting  some  of  the  commands  it  is 
possible  to  try  out  and  read  the  tool  tips  to  recall  what  is  the  correct  action  to  take. 
Dislikes:  Slow  to  load  data-  perhaps  good  to  have  a  progress  bar  so  that  user  knows  it’s 
still  working  on  it  (and  estimate  of  how  much  longer)  and  not  that  the  system  is  hanging, 
not  able  to  zoom  in  on  charts  (e.g.  to  see  just  the  2010  data  on  IEDs)-  making  it  hard  to 

read  off  and  count  the  values _ 

Dislikes:  Slow  as  hell.  Unresponsive.  Very  frustrating.  Shouldn’t  take  that  long  to 
gather  data.  Forgot  how  to  navigate  my  way  around  the  system.  Couldn’t  remember  how 
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Session  Three,  Question  Three:  What  were  your  likes  and  dislikes  with  your  third 

exposure  to  DaViTo? 

to  draw  a  chart. 


Likes:  I  remembered  most  of  the  main  features  and  did  not  have  to  be  retrained 
Dislikes:  Slow  system  response  times 

Likes:  Seemed  easier  to  use  by  the  third  trial.  Most  tasks  were  relatively  easy  to  recall. 
Dislikes:  Difficult  to  right  click  open  the  menu  for  the  data  chart. 

Dislikes:  Program  too  slow  to  respond 
Dislikes:  Loading  the  map  shapes  was  very  slow 

Likes:  It  took  a  few  minutes  to  remember  how  to  navigate  around  but  it  seemed  to  come 
back  to  me.  Good  user  interface. 

Dislike:  Is  the  program  is  a  little  slow  with  generating  charts. _ 

Like:  None. 

Dislike:  Not  using  the  map  control  bar _ 

Likes:  Ability  to  analyze  data  in  picture  format  to  gain  insights. 

Dislikes:  System  latency  in  loading  data.  Inability  to  know  whether  system  is  working 
or  not  i.e.  progress  bars  or  hour  glass  that  indicates  that  system  is  working  on  providing 
feedback  for  desired  input. _ 

Table  53.  Table  of  likes  and  dislikes  for  all  24  subjects  in  session  three.  Comments 
taken  directly  from  the  EyeWorks  software  with  no  modification. 


B.  FINAL  SURVEY 

Appendix  D,  part  B,  presents  additional  results  from  the  final  session  survey  free¬ 
form  questions.  Note  that  the  responses  in  Tables  54  through  64  are  not  altered  in 
content  from  what  the  subjects  entered  in  the  survey. 


_ Q:  (Effect_3)  Was  the  software  simple  to  use?  Please  explain. _ 

Some  of  the  finer  controls  associated  with  selection  suffered  due  to  lag. _ 

Yes. _ 

It  was  relatively  easy  to  use  but  chart  generation  seemed  more  complex  than  it  needed 

to  be. _ 

Yes-  with  the  exception  of  time  delays  without  status  icons  to  let  the  user  know  that 

infonnation  was  being  processed  by  the  system. _ 

Yes-  with  training _ 

The  basic  interface  is  simple  enough-  but  some  options  are  harder  to  find  than  others- 

therefore  they  are  more  difficult  to  remember  from  one  task  to  the  next. _ 

Yes-  mostly.  Again-  the  chart  output  and  geographical  labeling  were  issues.  Other 
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Q:  (Effect_3)  Was  the  software  simple  to  use?  Please  explain. 

than  that-  it  was  fairly  intuitive. 

Yes-  but  the  lag  created  errors  and  mistakes 

The  software  was  a  little  more  user  friendly  with  the  video  tutorial.  If  I  had  to  use 
DaViTo  without  the  tutorial- 1  would  have  been  lost! 

Generally  it  was  okay.  It  seemed  pretty  simple  at  first-  but  I  think  I  would  get  really 
frustrated  if  I  had  to  use  this  a  lot.  There  is  definitely  room  for  improvement. 

Yes.  Things  did  what  they  looked  like  they  would  do. 

After  three  sessions  it  became  easier  to  user  to  use. 

Yes-  the  software  was  easy  to  use.  Again-  the  delay  in  loading  the  map  data  was  the 
only  issue. 

It  was  relatively  intuitive  and  not  complicated  to  operate 

While  the  execution  of  commands  and  navigation  through  menus  was  simple  enough- 
the  analysis  features  (i.e.;  graphs)  were  sometime  difficult  to  interpret.  I  think  this 
could  be  solved  by  adjusting  the  formatting. 

Yes-  controls  are  intuitive  and  tooltip  pop-ups  were  helpful. 

No  it  sucked 

It  was  simple  to  use  but  it  could  definitely  be  improved  with  more  integration  of  map 
navigation  into  the  mouse  controls.  Separate  buttons  on  screen  for  panning-  zooming- 
grabbing-  and  data  selection  was  a  terrible  GUI  choice. 

For  the  most  part.  To  be  proficient  would  require  daily  use  in  order  to  not  forget  steps 
and  functions. 

Somewhat 

Fike  using  my  desktop  at  home 

Yes-  simple  user  interface. 

Simple-  yet  not  always  intuitive 

Mostly  intuitive.  More  use  would  have  made  it  fully  intuitive. 

Table  54.  Table  of  free-form  responses  to  question  Effect_3  for  all  24  subjects  in 

session  three.  Comments  taken  directly  from  the  EyeWorks  software  with  no 
modification. 


_ Q:  (Eff_4)  Was  the  visual  representation  easy  to  understand? _ 

Of  geographical  data  and  selections  -  yes.  Of  numerical  data  representations  -  no. 

Yes.  Something  to  consider  is  my  level  of  prior  training.  I  was  in  the  4th  Infantry 
Division  in  1998  as  we  were  testing  FBCB2.  So  I  have  had  a  lot  of  exposure  to  both 
that  and  BFT.  I  understand  most  of  this  set  up  from  those  two  systems.  The  charts 

configuration  was  new  to  me. _ 

Yes _ 

Sorting  of  the  data  points  within  a  data  set  with  different  colors  could  be  helpful. 

Yes _ 

Yes-  for  the  most  part.  Some  option  locations  were  inconsistent  with  other  similar 

mapping  GUIs.  That  makes  searching  around  for  it  difficult, _ 

Yes-  for  the  most  part.  The  charts  and  lack  of  geographical  labeling  were  really  the 
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Q:  (Eff_4)  Was  the  visual  representation  easy  to  understand? 

major  problem. 

Yes 

Yes 

If  we’re  talking  maps-  then  yes.  If  we’re  talking  graphs-  NO.  The  general  display 
wasn't  hard-  but  I  wouldn’t  say  it  was  really  easy  to  understand.  It  took  some  training 
to  get  used  to  it. 

Yes 

Yes  however-  labels  should  be  provide 

Yes-  no  issues  with  the  actual  visual  portion-  minus  the  charts. 

Relatively  so 

Of  the  maps  and  controls  yes-  but  not  the  charts  (as  I  stated  previously-  the  line  charts 
were  a  pain  to  read) 

Yes 

Yes 

Projecting  the  charts  directly  onto  the  regions  increased  clutter  on  the  screen  and  is  a 
bad  stylistic  choice.  The  map  projection  was  clunky  as  well-  and  the  software  would 
be  wise  to  use  a  Google  Earth  type  3d  view  vice  a  2d  projection. 

Yes.  Depending  on  the  level  of  detail-  labels  of  locations  would  be  useful. 

map-  yes,  graphs-  no 

Yes! 

Yes  except  for  some  of  the  charts  (the  cluster  charts  were  the  worst  I  think). 

Yes-  except  the  charts 

Yes 

Table  55.  Table  of  free-form  responses  to  question  Eff_4  for  all  24  subjects  in  session 
three.  Comments  taken  directly  from  the  Eye  Works  software  with  no 
modification. 


_ Q;(Use  2)  What  was  the  most  difficult  task  to  complete? _ 

Selection  of  individually  highlighted  areas  and  the  graphs  associated  was  sometimes 
difficult  due  to  the  need  to  "click  inside  -  wait  -  click  outside"  delays.  I  found  myself 

wondering  if  I  had  done  it  correctly  based  on  the  lag  in  response. _ 

Setting  up  the  charts.  I  definitely  needed  more  time  using  that.  I  know  that  was  a  focus 

last  session.  But  I  was  not  practiced  enough  to  remember  it  correctly. _ 

Graph  generation _ 

Reading  IED  attacks  in  2010  from  the  chart. _ 

Navigate  with  map  tools  hidden. _ 

Finding  the  chart  function  for  the  Kenyan  data. _ 

Reading  the  chart  output. _ 

Go  to  a  previous  view. _ 

Creating  the  charts _ 

Deciphering  the  graphs  to  get  usable  infonnation.  Way  too  difficult.  I'm  still  not  sure 
if  I  read  the  graphs  right. _ 
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Q:(Use_2)  What  was  the  most  difficult  task  to  complete? 

The  hardest  part  was  remembering  what  the  actual  question  was.  Everything  else 
seemed  pretty  straight  forward. 

None 

Hiding  the  map  controls. 

Reading  the  ridiculous  graphs 

Line  chart  interpretation  was  the  most  difficult  since  everything  is  so  closely  spaced 
and  you  have  to  look  over  to  the  left  to  judge  #  of  events. 

Reading  off  number  of  IED  attacks  (because  I  forgot  some  of  the  steps-  and  because 
the  chart  was  small) 

Reading  the  graphs. 

Setting  up  a  graph  for  specific  regions-  since  it  took  a  while  and  the  user  is  not  sure  if 
the  system  is  responding  or  still  waiting  for  an  input. 

Making  and  opening  the  chart. 

Finding  the  number  of  IED  attacks  in  Afghanistan 

creating  the  rectangle  shape 

Drawing  the  polygon  around  the  northwestern  provincial  district  to  generate  the  chart. 

Chart  selection 

Reading  the  graphs 

Table  56.  Table  of  free-form  responses  to  question  Use_2  for  all  24  subjects  in  session 
three.  Comments  taken  directly  from  the  Eye  Works  software  with  no 
modification. 


Q:  (Satisfaction  ?)  Were  you  satisfied  with  the  performance  of  the  software? 
_ Please  explain. _ 

I  would  like  to  know  when  it  is  processing  data  (to  know  to  wait).  If  analysis  tools 
(charts  and  data  displays)  were  offered  with  visual  examples  of  what  the  products 
looked  like-  it  might  assist  choosing  the  correct  one  for  the  job.  Otherwise-  the 
software  seems  like  it  would  be  a  reasonably  good  tool  for  both  analysis  and  visual 

display  of  data. _ 

Not  with  my  own  perfonnance-  no.  If  I  had  to  use  this  system  on  the  move- 1  would 
not  be  happy  with  my  current  level  of  training.  I  would  need  much  more  time  using 
the  system.  If  this  was  for  tactical  operations  center- 1  would  be  able  to  use  it  better. 

Be  able  to  also  input  data  easier. _ 

Based  upon  the  questions  posed  and  my  background  as  a  Surface  Warfare  Officer 
this  software  would  be  highly  useful  if  data  could  be  displayed  in  maritime  regions. 
With  this  software-  or  similar  software  and  good  historical  data  and  piracy  and 
counter  narcotics  missions  that  I  have  perfonned  could  have  been  effective  if  we  had 

the  data  and  tools  (software)  to  effectively  focus  our  limited  resources. _ 

Overall  it  has  a  lot  of  lag  and  lacks  fine  controls.  The  graphs  are  hard  to  read-  but 
there  may  be  functionality  to  make  them  more  readable.  With  some  improvement 

this  could  be  a  useful  tool  for  analysis  of  geographic  data. _ 

Yes-  but  more  training  is  needed  and  adjustments  to  the  software  would  be 
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Q:  (Satisfaction  ?)  Were  you  satisfied  with  the  performance  of  the  software? 

Please  explain. 

beneficial. 

Yes-  with  more  frequent  use  and  a  basic  help  menu-  this  would  be  useful  software. 

Somewhat.  I  feel  like  this  software  could  be  used  effectively  with  training  or  better 
software  design.  I  recommend  including  up-to-date  geographical  information  with 
the  software.  The  information  should  be  able  to  be  updated  automatically  and 
manually.  This  way-  it  could  be  updated  automatically  with  a  steady  internet 
connection  or  manually  if  on  deployment  or  in  the  field.  Also-  the  graph  display 
could  be  more  intuitive.  There  should  be  an  easier  way  to  change  the  display  of  the 
data  (line-  bar-  pie-  etc.)  by  right-clicking  on  the  displayed  chart.  The  analytical 
tools  that  are  included  should  be  addressed  in  training  or  in  the  user's  manual.  They 
should  be  written  so  a  person  with  a  sixth  grade  reading  (and  thinking)  level  could 
understand  them. 

Somewhat.  If  the  'dislikes'  were  fixed-  it  would  be  very  helpful. 

I  feel  the  software  is  a  great  tool-  however-  without  a  larger  knowledge  base  of  how 
to  run  the  software- 1  would  be  at  a  disadvantage. 

No-  I  had  the  worst  time  deciphering  the  graphs.  I  feel  that  I  could  manipulate 
DaViTo  well  enough  to  get  the  graphs-  but  it's  ridiculous  to  get  any  usable  data  from 
the  graphs. 

No-  too  slow.  And  in  general  the  graphs  aren’t  very  good.  Seems  to  be  too  much 
information  at  once. 

Moderately  satisfied-  while  the  tool  is  simpler  to  use  that  ArcGIS  it  is  very  limited 
requiring  improvements  such  as  user  interface  improvements  to  provide  feedback  to 
the  user-  including  labels  for  maps-  improving  system  latency-  etc. 

I  did  not  like  the  amount  of  time  that  it  took  to  load  the  data  sets.  If  there  was  a 
notification  that  stating  loading-  then  maybe  it  would  have  been  ok. 

Same  comments  as  previous.  Relatively  easy  to  use  software.  I  could  pick  it  up 
where  I  left  off  each  session  with  only  a  few  seconds  of  menu  searching  at  the  most 
to  re-familiarize  myself  with  the  tools.  The  graphs-  however-  are  mind-numbingly 
bad.  They  are  difficult  to  read.  If  they  were  as  intuitive  as  the  basic  controls  for  the 
rest  of  the  software-  I  would  recommend  this  program  wholeheartedly.  As  it 
currently  stands-  I  would  break  my  monitor  trying  to  read  even  a  small  number  of 
them. 

I  can  definitely  see  potential  in  the  analysis  features  of  this  software-  as  long  as  the 
data  is  accurate  and  timely-  but  the  perfonnance  appeared  a  little  slow  to  me  (not 
knowing  what's  under  the  hood).  I  understand  I  could  create  smaller  sets  of  data- 
but  this  might  be  counterproductive  and  not  provide  the  big  picture  depending  on  the 
intent/scope  of  this  software. 

It  was  adequate.  Somewhat  slow-  but  that  is  understandable  considering  the  amount 
of  data  involved.  Functionalities  are  quite  intuitive.  Inability  to  zoom  into  charts  is 
slightly  inconvenient. 

No.  It's  jittery-  unresponsive.  Takes  forever  to  load  infonnation.  The  user  help  is 
not  very  friendly.  This  software  needs  serious  help. 

While  the  goals  of  the  software  and  tools  available  were  helpful-  I  felt  that 
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Q:  (Satisfaction  ?)  Were  you  satisfied  with  the  performance  of  the  software? 

Please  explain. 

functionality  could  be  improved  on  the  map  interface.  It  is  slow  and  clunky-  with 
non-intuitive  pan  /  zoom  /  selection  buttons.  Support  for  mouse  wheel  scroll  for 
zooming  would  be  very  helpful-  and  reducing  the  delay  between  data  selection  and 
analysis  would  be  helpful  as  well. 

The  software  yes.  The  chart  interpretation  no. 

Not  really-  it's  very  slow  to  respond  to  commands. 

Yes  very  satisfied-  I  accidently  marked  answers  strongly  disagree  when  I  meant 
strongly  agree.  This  software  is  user  friendly  and  not  difficult  to  navigate  around  in. 
The  time  required  to  load  and  unload  shapes  had  quite  a  bit  of  a  delay. 

The  software  has  some  good  output.  The  type  of  charts  needs  to  be  looked  at-  some 
charts  are  better  than  others  and  easier  to  read.  The  speed  of  the  program  was  a  bit 
slow  at  times  generating  charts. 

Series  of  clicks-  left  or  right  and  where  you  had  to  click  was  not  straight  forward 

I  assume  that  the  graphs  can  be  made  larger.  Latency  of  data  display  is  not  good.  If 

I  was  using  this  in  an  operational  environment  I  would  have  to  get  up  and  smoke  a 
cigarette  to  kill  the  time  while  waiting  for  data  to  load.  If  I  did  not  have  a  smoke  I 
would  just  beat  my  machine  with  a  hammer.  I'm  confident  that  once  the  data  is 
displayed  that  I  would  be  able  to  use  the  tool  positively. 

Table  57.  Table  of  free-form  responses  to  question  Use  2  for  all  24  subjects  in  session 
three.  Comments  taken  directly  from  the  Eye  Works  software  with  no 
modification. 

Q:  (Improve  3)  How  would  you  improve  DaViTo?  Please  explain. 

There  is  something  off  with  the  mouse  selection  of  individual  territories  associated  with 
lag  and  the  need  to  deselect  an  area  prior  to  the  next  selection.  Perhaps  a  "waiting" 
icon  (hourglass?)  could  be  displayed  to  let  users  know  when  to  wait  prior  to  entering 
the  next  command  would  be  appropriate. 

More  hands  on  training.  Executing  the  tasks  that  are  necessary  to  use  the  software 
efficiently. 

I  would  add  chart  generation  to  the  toolbar. 

Improved  lag  on  map  controls-  status  icons  for  processes. 

Progress  icons-  more  training 

Find  the  most  commonly  used  geographic  analysis  software  out  there  that  resembles 
DaViTo  and  organize  the  GUI  to  resemble  that  as  closely  as  possible. 

See  previous  comments. 

Improve  the  lag  problem. 

I  would  improve  DaViTo  with  a  tutorial  by  an  onsite  representative  to  teach  the 
functionality  of  the  program.  I  would  also  leave  a  basic  video  tutorial  to  ensure  that  the 
use  of  the  system  was  driven  home  by  the  user. 

If  you  hover  over  an  item  it  should  pop  up  "quickly"  telling  you  what  it  is.  It  seemed  to 
be  pretty  delayed.  I  was  still  doing  a  lot  of  'trial  and  error’  while  manipulating  the 
software  in  my  third  session.  I’d  also  have  a  user-friendly  help  section  that  we  could 
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Q:  (Improve  3)  How  would  you  improve  DaViTo?  Please  explain. 

click  on.  Additionally-  I’d  like  to  see  a  way  to  personalize  the  display  buttons  etc.  for 
use.  That  way  I  could  put  things  where  I’d  like  them. 

Make  it  faster. 

Improve  user  interface  issues  essentially  to  make  it  more  intuitive  for  the  user;  improve 
the  flexibility  so  the  user  can  modify  output;  and  fix  system  latency. 

Improve  the  data  set  loading  issue.  Other  than  that-  the  tool  was  excellent. 

Make  the  graphs  easier  to  read 

Improved  charts-  fix  the  lag  (see  previous  comments) 

Navigation  button  could  be  renamed  'zoom  box'  or  something  similar  to  avoid 
confusion  with  the  map  navigation  button.-  user  could  be  given  more  control  of  the 
charts  (e.g.  zoom  in  to  some  portion  of  the  data)-  loading  time  if  possible  (e.g.  when 
loading 

Make  it  more  user  friendly  and  responsive. 

Comments  provided  in  previous  questions.  Better  maps  and  response  times- 
essentially. 

Spend  more  time  accomplishing  the  tasks  and  improving  the  user  friendly  charts. 

Speed  it  up  and  improve  the  graphical  outputs 

Figure  a  way  to  remove  the  delays  when  loading  or  creating  shapes 

Again-  improve  some  of  the  charts-  I  thought  the  cluster  charts  were  not  good  visual 
representations  of  data. 

Charts 

Table  58.  Table  of  free-form  responses  to  question  Improve_3  for  all  24  subjects  in 

session  three.  Comments  taken  directly  from  the  EyeWorks  software  with  no 
modification. 


Q:  (Improve  4)  What  was  the  most  frustrating  feature  of  DaViTo? 

The  lag. 

On  screen  help.  I  should  have  used  the  help  button  more.  This  would  likely  be 
eliminated  by  more  training. 

Sluggishness  and  chart  generation. 

Lag 

Not  knowing  if  it  was  frozen  or  thinking 

Finding  the  chart  option  for  Kenya. 

Chart  output  was  difficult  to  match  to  months  and  difficult  to  add  up  events. 

Lag  time  in  loading  things  and  lag  time  between  selection  and  reaction. 

Interpreting  charts  and  lag  time  for  map  loading 

It  seemed  really  slow.  Lots  of  delay  loading  maps  and  right  clicking.  Sometimes  the 
delay  would  be  counterintuitive  because  I  thought  I  was  doing  something  wrong-  but 
in  reality  it  was  just  slow. 

It  was  slow. 

System  latency 

Data  set  loading  time. 
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Q:  (Improve_4)  What  was  the  most  frustrating  feature  of  DaViTo? 

Graphs-  graphs-  graphs 

Lag  time  when  selecting  maps/data  sets 

Load  times 

Slow  and  unnecessarily  complicated  to  navigate  way  around  data.  Confused  on  where 
to  look  to  load  data. ..took  too  much  time. 

Slow  map  response  times. 

Interpreting  the  charts. 

Sluggish  responses 

The  delay  experienced  when  creating  or  loading  a  shape 

The  slow  time  of  generating  charts  and  trying  to  read  and  interpret  the  cluster  charts. 

Which  mouse  button  to  use  and  when  in  what  context 

Latency  of  display/function  execution 

Table  59.  Table  of  free-form  responses  to  question  Improve_4  for  all  24  subjects  in 

session  three.  Comments  taken  directly  from  the  EyeWorks  software  with  no 
modification. 


Q:(Improve  5)  What  did  you  most  like  about  DaViTo?  Please  explain. 

Like  ARCGIS-  the  ability  to  display  data  on  a  map  is  very  useful.  I'm  not  sure  how 
this  software  would  be  better  than  ARCGIS-  though-  and  that  is  a  widely 
disseminated  and  developed  product  that  is  significantly  less  buggy.  The  ability  to 
predefine  areas  that  a  user  could  then  open-  view-  and  process  for  analysis  makes  it 
possible  to  share  work  without  having  to  recreate  it  while  using  it  for  different 
purposes.  That  could  be  a  great  potential  time  saver  that  would  also  reduce 
confusion  between  units  by  sharing  world  views. 

The  set  of  the  map  with  controls  to  the  left.  Seemed  much  more  logical  and  I  could 
see  more  of  the  area  I  wanted  to  see. 

Visual  display  of  event  data  and  layered  maps. 

Good  visual  representation  of  geographic  data. 

I  like  maps 

Overlay  of  chart  data  by  geographic  region  so  you  can  eyeball  outliers/trends. 

The  general  interface.  It  was  intuitive. 

Simple. 

I  liked  that  basic  graphing  infonnation  could  be  loaded  to  represent  trend  data  for  a 
given  location. 

It  seemed  simple.  It  didn't  look  overwhelming  or  intimidating.  But  in  the  end 
simplicity  wasn’t  the  best  way  to  go. 

Nothing. 

That  it  is  open  source  and  can  be  improve 

The  ability  to  populate  different  districts  and  boundaries  for  different  countries/areas. 
The  ability  to  select  a  specific  region-  and  then  it  provide  detailed  charts  on  the  data 
is  a  great  benefit  as  well. 

Ease  of  use 
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Q:(Improve_5)  What  did  you  most  like  about  DaViTo?  Please  explain. 

Fairly  intuitive. _ 

Gives  a  good  view  of  the  whole  data  set  at  a  glance _ 

Clicking  on  a  data  set  and  seeing  data _ 

Integrated  data  analysis. _ 

Navigation  of  the  map  and  the  ability  to  bring  up  the  loaded  data  about  the 

provinces. _ 

Nothing  really  excited  me _ 

Easy  to  use _ 

Simple-  easy  user  interface.  I  am  not  usually  fast  with  learning  computer  and/or 
software  programs  and  the  fact  that  I  learned  some  basic  functionality  with  this 

program  in  a  short  amount  of  time  speaks  well  about  its  usability. _ 

Graphical  display _ 

Ability  to  visualize  data _ 

Table  60.  Table  of  free-form  responses  to  question  Improve_5  for  all  24  subjects  in 

session  three.  Comments  taken  directly  from  the  EyeWorks  software  with  no 
modification. 


Q:(Improve_6)  What  would  you  change  about  the  color  scheme  /  page  layout? 

Please  explain. 

Slightly  greater  contrast  might  make  it  more  comfortably  viewable-  though  it  was 
sufficient. 

For  daytime  use-  the  color  scheme  was  sufficient.  The  major  change  would  be  for 
detailed  use  having  more  area  relief.  This  would  be  useful  both  on  the  move  and  in 
planning.  I  would  know  how  to  plan  for  movement  times. 

Nothing 

Nothing 

Nothing 

Nothing 

No. 

Nothing. 

I  would  add  actual  colors-  vice  the  drab  grey.  I  would  also  have  areas  of  high 
importance  be  color  coated  differently  than  the  surrounding  areas  to  represent  their 
importance. 

Too  bland.  I  don’t  need  a  lot  of  colors-  but  if  I  had  to  look  at  it  all  the  time  I'd  like  it  to 
be  easier  to  customize.  The  default  colors  scheme/page  layout  should  be  a  little  less 
bland.  I  get  bored  just  looking  at  it. 

Less  shades  of  blue. 

Given  that  used  the  tool  for  less  than  2  hours-  I  cannot  properly  recommend  a  color 
scheme  or  page  layout 

No 

Nothing  of  note 

Kind  of  bland  compared  to  most  applications.  A  little  more  vibrant  color  scheme  and 
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Q:(Improve_6)  What  would  you  change  about  the  color  scheme  /  page  layout? 

_ Please  explain. _ 

larger  fonts  might  make  it  more  attractive.  Compared  to  dynamic  mapping  software 
(like  Google  Earth)  -  this  is  at  the  lower  end  of  the  spectrum  (not  like  Apple  user 

interface  that  "slides"  the  map  along  and  incorporates  inertia). _ 

It’s  fine  the  way  it  is.  perhaps  the  default  color  when  loading  a  data  set  could  be 

something  of  high  contrast-  e.g.  red-  to  start  with  (instead  of  dark  grey) _ 

Use  colors  to  depict  certain  areas  on  the  screen  for  better  navigation. _ 

Layout  was  good-  but  utilizing  a  3d  map  like  Google  Earth  would  be  far  more  intuitive 

and  helpful. _ 

No  issues. _ 

Nothing _ 

Nothing _ 

None _ 

Nothing _ 

I  think  it  is  fine. _ 

Table  61.  Table  of  free-form  responses  to  question  Improve_6  for  all  24  subjects  in 

session  three.  Comments  taken  directly  from  the  EyeWorks  software  with  no 
modification. 


Q  :(Improve_7)  How  would  you  improve  the  mouse  interaction  with  the 

software?  Please  explain. 

I  don’t  believe  there  were  positive  indications  of  mouse  selection  or  actions  associated 
with  some  of  the  functionality.  When  specific  selection  is  required-  definitive 
highlighting  might  be  possible. 

Nothing. 

Unknown. 

No  improvement  necessary 

Right  click  on  map  gives  all  operations 

Nothing 

I  wouldn’t.  It  was  fine. 

Nothing 

No  comment 

Just  make  ’right  clicks'  faster.  Also-  give  more  ways  to  navigate  with  just  the  mouse 
like  panning.  I  think  there  is  that  option  but  it  was  too  slow. 

I  wouldn’t 

It  seems  that  the  mouse  interaction  works  properly 

The  mouse  is  fine. 

It  worked  fine 

Nothing  -  works  just  as  expected  with  left  and  right  clicks  for  primary  and  secondary 
actions. 

Mouse  clicks  for  the  distance  tool  was  a  bit  confusing 

Scroll  map  while  depressing  the  left  mouse  key 
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Q  :(Improve_7)  How  would  you  improve  the  mouse  interaction  with  the 
_ software?  Please  explain. _ 

Integration  of  map  navigation  into  mouse  buttons. _ 

Change  how  the  chart  menu  option  is  brought  up  by  clicking  on  the  location.  Seemed 

to  take  numerous  attempts  for  the  menu  to  pop  up. _ 

No  issues  with  the  mouse _ 

None _ 

Table  62.  Table  of  free-form  responses  to  question  Improve_7  for  all  24  subjects  in 

session  three.  Comments  taken  directly  from  the  EyeWorks  software  with  no 
modification. 


Q  :(Improve_8)  How  would  you  like  to  see  the  software  employed?  Please 

explain. 

If  this  software  could  be  adopted  by  intel  and  ops  analysis  groups-  it  would  be 
awesome  to  share  prepared  data.  That  said-  it  would  have  to  be  unilaterally  adopted 
within  the  joint  construct  or  it  would  be  a  waste  of  time  because  data  portability  would 
suffer. 

In  TOCs  to  assist  planning. 

Resource  utilization  based  upon  event  data.  Allows  focused  use  of  limited  resources. 

Data  analysis  for  counter  piracy  operations 

Intel-  mission  planning 

I  could  have  used  this  to  track  my  squadron’s  IED  detection  performance.  P-3C 
missions  up  and  down  Iraq’s  supply  routes  report  hundreds  of  "hot  spots"  and  some  get 
investigated  while  some  do  not.  Those  that  are  investigated  may  be  IEDs-  but  many 
more  are  false  positives.  The  trends  are  somewhat  geographically  related-  but  the  data 
isn’t  easy  to  visualize.  Better  data  tracking  would  be  required  to  use  DaViTo  for  this- 
but  if  you  had  it-  you  might  get  the  commands  to  track  data  appropriately. 

I  can  think  of  several  uses.  It  would  be  useful  for  anything  from  intelligence  events  to 
liberty  incidents  to  logistical  tracking.  Wide  variety  of  possible  uses. 

Not  sure. 

I  know  this  is  mainly  an  Army  software  tool-  but  if  there  could  be  incorporation  of  it 
among  other  branches  of  military  it  could  broaden  the  horizon  of  data  available. 

I'm  not  sure... 

No  idea. 

Currently  my  experience  with  the  tool  is  exploring  event  data  but  this  tool  could  easily 
be  modified  and  employed  for  sustainment  and  logistics  personnel. 

In  training  and  field  environments  (Afghanistan-  of  course) 

I  suppose  if  it  were  made  available  for  basic  research.  I  had  never  heard  of  it  before. 

Integrate  this  software  into  CPOF  rather  than  as  a  stand-alone  application.  The 
features  I've  seen  indicate  a  focus  on  analyzing  data  sets  for  trending  that  could  be 
applicable  to  both  the  tactical  and  operational  levels-  so  it  would  be  better  to  have  a 
shared  platform  for  looking  at  the  same  data. 

Probably  useful  for  the  OR  and  data  analysis  guys. 
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Q  :(Improve_8)  How  would  you  like  to  see  the  software  employed?  Please 
_ explain. _ 

Used  for  trend  analysis  in  order  to  make  better  decisions  at  company  level  and  higher 

Basic  analysis  of  available  data  for  military  needs. _ 

Useful  as  a  planning  tool  for  events  in  areas  being  entered.  Could  give  a  visual 

indication  of  the  activity  expected  to  be  encountered. _ 

Could  be  used  in  a  HQ  to  conduct  trend  analysis  for  SIGACTS _ 

Would  be  useful  for  land  and  see  navigation _ 

It  seems  it  would  be  more  useful  for  land  based  forces.  I  serve  on  Navy  ships-  and  I 

do  think  it  would  add  much  value  there. _ 

I  have  no  reference _ 

Would  be  useful  in  organizational  headquarters  first  in  a  garrison  setting  and  then 
deployed.  Garrison  use  would  ensure  that  both  the  tool  and  the  user  were  familiar 
with  the  system  and  the  interpretation  of  the  data  prior  to  operational  employment. 

Table  63.  Table  of  free-form  responses  to  question  Improve_8  for  all  24  subjects  in 

session  three.  Comments  taken  directly  from  the  EyeWorks  software  with  no 
modification. 


Q:  (Visual_2)  What  more  would  you  like  to  see  in  the  output  of  this  software? 

If  you  could  link  the  data  through  R-  Jump-  or  Excel  -  the  displays  might  be  more 
consistent  with  some  of  the  data  display  tools  we  all  have  to  use  anyway  (especially 
when  constrained  to  an  NMCI  machine).  Perhaps  both  cumulative  and  incremental 

charts  could  be  displayed  simultaneously  (to  alleviate  confusion). _ 

Expanded  graphing  such  that  you  could  zoom  into  a  graph  and  discern  individual  data 
points  -  including  scales  that  would  be  appropriately  detailed  to  discern  where  the 

data  points  sit. _ 

Just  a  clear  definition  of  whether  events  are  independent  or  cumulative.  Also-  why  do 
the  events  have  to  be  stacked  or  put  together?  This  is  not  provide  me  with  more 
information.  If  I  just  wanted  shootings  and  not  IEDs-  why  would  I  get  all  the 

information  all  at  once?  I  personally  want  to  be  able  to  tailor  the  information. _ 

User  modifiable  graphs. _ 

Statistical  analysis  of  different  portions  of  the  data-  i.e.  analysis  of  all  IED  attacks  for 

a  particular  month  over  all  of  the  provinces. _ 

Data  should  be  more  clear  in  its  representation.  Possible  hovering  mouse  shows 

nearest  data  point  value _ 

More  options  for  graphing  than  are  shown  in  the  questions  (assuming  that's  all  there 
is).  The  axis  labels  are  confusing  and  some  of  the  cumulative  line  graphs  are 

misleading  because  of  their  alignment. _ 

I  addressed  this  a  bit  already.  Make  the  manipulation  of  the  output  more  intuitive.  It 
should  be  able  to  be  manipulated  by  right-clicking  or  with  first-level  menu  options. 
Perhaps  a  GUI  where  the  user  could  ask  specific  questions-  like  posed  in  this 
experiment-  and  the  software  displays  the  correct  answer  in  addition  to  the  graph. 
This  could  be  done  from  pull  down  menu  selections.  Many  current  financial  websites 
do  this  for  the  customer. 
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Q:  (Visual_2)  What  more  would  you  like  to  see  in  the  output  of  this  software? 

Basic  examples  on  how  to  read  the  data  would  be  beneficial. 

A  way  to  hover  over  the  graphs  and  get  data  points  (amounts)  would  help.  Or  be  able 
to  click  on  the  graph  and  get  the  actual  numbers  instead  of  generalizations  done  with 
the  graphs. 

For  big  graphs  it  is  hard  to  see  where  the  values  line  up  on  the  y  -axis.  If  the  axis 
could  be  slide  to  the  right  so  it  is  next  to  the  data  that  one  is  interested  in  it  would 
make  interpreting  the  data  easier.  Also-  it  would  be  nice  to  zoom.  But  all  of  this  will 
be  pointless  if  the  program  keeps  running  so  slow.  Instead  of  redrawing  all  the  lines 
frame  cycle-  like  I  know  this  thing  is  doing-  why  not  write  the  data  to  a  background 
buffer.  Stuff  can  be  put  over  this  buffer  but  it  doesn’t  need  to  be  recalculated  every 
time. 

Essentially  improve  flexibility  so  the  user  can  modify  and  edit  output  similar  to 
capabilities  provided  by  other  tools  including  excel. 

Nothing  more  to  add;  just  improve  the  output  of  some  of  the  charts-  meaning  the 
actual  display  (size  in  general). 

Make  the  graphs  more  intuitive-  as  the  software  itself  is. 

Have  the  ability  to  toggle  quickly  between  the  types  of  charts  to  pick  which  one 
presents  the  most  clear  summary  of  events  for  which  you  are  analyzing.  It  would  be 
interesting  to  add  some  real  analysis  features  such  as  extrapolation-  prediction-  and 
so  forth  to  leverage  statistics-  but  I'm  not  sure  how  difficult  that  would  be. 

Bigger  charts  (when  required) 

Graphs  that  a  human  can  understand  without  performing  some  kind  of  advance  math 
problem. 

Explanations  available  for  users  that  are  not  proficient  with  basic  statistical  analysis 
would  be  useful  in  the  output. 

More  user  friendly  charts. 

The  graphs  should  be  tailored  to  the  query. 

Nothing  else 

Easier  to  read  charts. 

Pie  charts-  for  one 

Cluster  column  format  needs  to  be  spread  out.  Line  the  data  up  with  the  grid  line 
references  to  make  it  easier  to  follow. 

Table  64.  Table  of  free-form  responses  to  question  Visual_2  for  all  24  subjects  in 

session  three.  Comments  taken  directly  from  the  EyeWorks  software  with  no 
modification. 
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APPENDIX  E.  TASK  WORKSHEET 


Appendix  E  contains  the  list  of  tasks  for  the  final  session,  as  shown  in  Table  65. 
This  form  was  used  by  the  experiment  session  proctors  to  track  subject  performance.  The 
mouse  clicks  and  times  for  each  task  were  recorded  by  EyeWorks.  The  proctor 
monitored  and  verified  the  status  of  satisfactory  task  completion. 


Participant  Number 

Task 

Clicks 

Time 

Completed 

Y/N 

1.)  Open  WITS. csv 

l.A  Left  click  +  button  in  Data  Set 

l.B  Left  click  WITS. csv 

l.C  Left  clicks  WITS. csv  open  button 

l.D  Left  click  load  button 

Total 

2.)Load  Kenya_Roads  Shapefile 

2. A  Left  click  +  button  in  Predefined 
Polygons  box 

2.B  Left  click  Shapefile  Kenya 

2.C  Left  click  open 

Total 

3.)  Go  to  Africom  view 

3. A  Left  click  Views 

3.B  Left  click  Africom 

Total 

4.)  Create  a  rectangle  chart  on  the  data  Point 
in  the  southwestern  most  part  of  Kenya  and 
display  the  chart 

4. A  Right  click  on  screen  to  create 

4.B  Left  click  on  rectangle 

4.C  Left  or  right  click  to  draw  rectangle 

4.D  Left  click  on  screen 

4.E  Right  click  on  rectangle 

4.F  Left  or  right  click  on  "Display  Chart" 

Total 
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5.)  Close  and  Delete  Chart 

5. A  Left  click  red  x  Button 

5.B  Right  click  on  rectangle 

5.C  Left  or  right  click  on  delete  chart  or 
delete  all  charts 

Total 

6.)  Hide  Map  controls 

6. A  Right  or  left  click  on  control  tab 

6.B  Right  or  left  click  on  hide  tool  panel 

Total 

7.)  Navigate  to  Iraq  without  map  control  panel 

Option  1 

7. A  Right  or  left  click  control  Panel 

7.B  Right  or  left  click  on  pan 

7.C  Navigate  to  Iraq 

Total 

Option  2 

7. A  Right  or  left  click  views 

7.B  Right  or  left  click  on  Iraq 

Total 

8.)  Go  back  to  previous  view 

8. A  Right  or  left  click  control  tab 

8.B  Right  or  left  click  display  tool  panel 

8.C  Left  click  go  back  previous 
projection  icon 

Total 

9.)  Remove  all  the  Data  sets  and  Shapefiles, 
load  the  lndure_Demo  data  and 
afganistan_provincial_boundaries  Shapefile 

10.)  Now  please  answer  the  following 
question:  what  were  the  total  events  for  IED 
attacks  for  2010  in  the  Northwestern  most 
provincial  boundary? 

11.)  Does  it  appear  that  IED  attacks  are  on  the 
rise  from  2006  in  the  same  provincial 
boundary? 

Total 

Table  65.  Worksheet  used  by  experiment  proctors  for  data  recording.  Includes  all  tasks 
for  session  three. 
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APPENDIX  F.  APPROVED  IRB  DOCUMENTS 


Appendix  F  contains  the  IRB  approval  cover  sheet  and  author  generated  IRB 
documents.  This  includes  the  consent  form,  call  for  participants,  demographic  survey, 
and  question  list  for  the  final  survey.  These  are  the  approved  documents  from  the  IRB 
for  use  in  this  thesis. 


Naval  Postgraduate  School 

Human  Research  Protection  Program 


I  n. tin  President.  Naval  Postgraduate  School 
Via.  Chairman.  Institutional  Revie*  Board 
To  l)r  J»  Hyun  Yang  MOVES  Institute 
Mr.  lease  Huston.  MOVES  Institute 
MAI  James  Henry.  TRAC  Monk-ivy 
LT  l>fcr  Rmbimwiw  l  *SN 
L T  Darren  Cutler.  USN 

SUBJ:  USABILITY  TESTING  AND  WORKFLOW  ANALYSIS  OP  IRE  TRADOC  DATA 
VISUALIZATION  TOOL  (DAVITO) 

Enel:  (11  Approved  IRB  Protocol 

1  The  NPS  IRB  is  pleased  to  inform  you  that  the  NPS  President  lias  approved  your  project 

NFS  IRB  i  M  t  he  approved  IRB  Protocol  is  found  in  enclosure  1 1  y 

Conviction  of  the  Cl  II  Research  Ethics  Training  has  been  confirmed. 

2  I  bis  approval  expires  on  .10  September  2012.  If  additional  time  b  required  to  complete  the 
research,  a  continuing  revie*  report  must  he  approved  by  the  IRB  and  NPS  President  prior  to  the 
expiration  of  approval  At  expiration  all  research  (subject  recruitment,  data  collection,  analysis 
ol'duta  containing  PI  l>  must  cense. 

3.  You  are  required  to  report  to  the  IRB  any  unanticipated  problems  or  serious  adverse  events  to 
the  NPS  IRII  within  24  hours  of  the  occurrence 

4  Any  proposed  changes  in  IRB  approved  research  must  be  reviewed  and  approved  by  the  NPS 
IRB  and  NPS  President  prior  to  unplemcntabon  except  where  necessary  to  eliminate  apparent 
immediate  hazards  to  research  participants  and  subjects. 

.V  As  the  Principal  Investigator  it  is  your  responsibility  to  ensure  iliac  the  research  and  the 
actions  of  all  project  personnel  involved  in  cunducting  this  study  will  cunlonn  with  the  IRB 
approved  protocol  and  I  KB  requircmenrVpol*:»ci. 

6  Aflcr  the  ex|x*njnenl  is  completed  the  Principal  Investigator  will  submit  to  the  Human 
Subjects  Protection  Office,  all  signed  informed  consent  documents,  unanticipated  problem 
reports,  adverse  event  reports  and  a  End  of  Experiment  Report  The  Human  Subjects  Research 
Office  will  secure  these  documents  far  10  years  and  then  forward  to  the  neatest  FRC 

Darnel  T  Oliver 

President 

Naval  Postgraduate  School 


CAPT JofnK  Schmidt.  MSC  USN 
Institutional  Review  Board 


APR  1  i  2012 
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Suh|ect  Line  -  Call  for  Participants  in  MOVES.TRAC  Monterey  Student  Thesis  Research 

Please  join  our  study  as  a  volunteer  for  the  following  thesis 
*°  USABILITY  TESTING  AND  WORKFLOW  ANALYSIS  OFTHE  *** 
"*  TRADOC  DATA  VISUALIZATION  TOOL  (DaViT*)  *•* 

Dear  Prospective  Participant. 

Would  you  like  to  participate  tn  NPS  student  thesis  research  to  improve  a  geospatial 
mapping  tool  for  use  by  the  U.S.  military?  We  arc  NTS  students  in  the  Modeling.  Virtual 
Environments,  and  Simulation  (MOVES)  Institute  conducting  a  usability  test  as  part  of  our 
thesis.  The  purpose  of  the  research  is  to: 

1 .  Evaluate  the  usability  and  efficiency  of  the  DaViTo  ami  determine  how  closely  the  actual 
software  workllow  reflects  how  the  user  thinks  of  the  workflow. 

2.  Determine  how  the  DaViTo  efficiency  and  usability  con  be  improved?  What  design 
suggestions  need  to  be  made  to  achieve  these  improvements'* 

This  research  opportunity  u  open  to  all  NPS  military  students  and  civilians  who  are  not 
color  blind  and  were  not  involved  in  the  development  of  the  DaViTo  software. 

Student  participation  is  crucial  to  this  thesis  and  meeting  our  graduation  requirements, 
and  is  completely  voluntary.  There  is  no  direct  benefit  to  the  subject  The  research  will  take 
place  in  W'atkins  2I2B  over 3  sessions  approximately  one  week  apart,  beginning  and  ending  on 
dates  TBD.  The  sessions  will  take  approximately  it)  on  minutes 

Should  vnu  choose  to  participate  wc  will  ensure  your  performance  and  personally 
identifying  information  (Pll)  w  ill  be  secure.  Collective  performance  data  and  survey  results  will 
he  used  in  our  final  analysis  and  puhlicatioci.  however  no  Pit  data  will  be  included  and  has  no 
hearing  on  our  analysts.  We  will  lake  die  following  precautions  to  protect  participants: 

I  All  email  correspondence  will  be  via  secure  NPS  email. 

2.  No  digital  files  containing  Pll  will  be  saved  to  our  computers. 

3.  Any  maicrials  printed  will  Itave  all  Pll  removed  and  will  be  securely  locked  in  a  cabinet. 


Thank  you  in  advance  tor  your  consideration  and  participation.  Please  reply  fo  this  email 
to  setup  your  participation.  Your  assistance  is  personally  important  to  us  as  we  work  to  complete 
our  graduation  requirements. 

Sincerely,  US  III 

aKWSb 

units 

SEP  3fl  2012 
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Darren  CiKlcr 
LT  USN 


Tvlcr  Rasmussen 

LT  USN 


IRB  Chau  Contact  InformatKWi 
CAPT  John  K.  Schmidt.  USN 
ikschmidii;  nns  ohi 
831-656-3864 


IPS  IPI 

4fPiov(a 
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Nuvnl  Postgraduate  School 
Consent  to  Participate  in  Research 


Introduction 

V'csi  arc  mvited  to  participate  ill  a  research  study  tided  'livability  Testing  and  Workflow  Anah  it*  ot  the 
TRADOC  Data  Visualization  Tool  (DaViTo)"  This  research  is  focused  on  evaluating  the  usability  and 
workflow  of  the  DaViT o  tool,  with  an  aim  of  improvement. 

Purpose: 

The  purpose  of  the  reward)  Is  to  provide  recommendations  for  improvement  on  DaViTo  from  user  uipurs 
and  statistical  information  collected  during  human  to  software  interaction  session*.  Thaw  leccmroendatiom 
arc  intended  to  improve  the  workflow  ,  usability  and  visual  clarity  of  DaViTo. 

Procedures 

We  are  asking  military  officers  and  civilians  to  participate  in  a  usability  study  of  the  Dura  Visualization 
Tool  (DaViTo)  developed  hy  TRAC-Moeilerey .  The  target  number  of  participants  is  24.  The  study  will  be 
oomhrcttd  over  a  three  to  four  week  period  During  the  study  we  will  use  comet  at  and  software  to  estimate 
and  record  eye  gaze  direct  ion.  eye  dwell  time,  and  mouse  and  keyboard  inputs  These  inputs  will  be 
recorded,  but  the  indiv  whiiil  participant  will  not  be  recorded  The  participants  will  be  asked  to  attend  two 
familiarization  and  training  sessions  that  will  last  ior  approximately  Ml  -  M  ill  mutes,  and  will  be  separated 
by  a  one  week  pxTiod.  then  a  final  evaluation  session  I  n  the  first  training  and  fnntiliarizaf  ion  session  the 
participant  w  ill  he  asked  to  complete  a  demographic  questionnaire  and  create  a  profile  in  the  eye  tracking 
software,  then  watch  a  familiarization  video  of  the  DaViT o  software  During  the  second  trainmg  and 
fam  ilmrization  session  the  participant  w  ill  watch  a  v  idco  that  walks  through  common  tasks  DaViTo  was 
designed  to  handle  the  third  session  will  consist  of  the  experiment,  where  the  participant  will  be  asked  to 
complete  tasks  in  DaViTo  similar  to  the  ones  covered  in  the  training  and  familiarization  sessions.  During 
this  session  the  parlvcipnnts'  eye  gjute.  eye  dwell  time,  and  mouse  and  keyboard  inputs  will  be  recorded  for 
usability  analysis.  When  the  participant  hns  completed  the  DaViTo  tasks,  they  will  be  asked  to  complete  a 
survey  on  DaViTo  about  the  general  sUisfactui  of  the  tool  and  opinions:  on  visual  formats.  The  experiment 
will  take  place  one  week  after  the  second  training  and  familiarization  session,  and  last  for  *>praximalcly  61) 
minutes.  We  w  ill  provide  information  on  the  study  results  if  desired-  The  entire  study  should  take 
approximately  1  to  4  hours  Aliy  inlcrmatian  will  be  used  only  for  this  research  endeavor. 

Voluntary  Nature  of  (be  Study 

V  our  participation  in  this  study  is  strictly  voluntary.  If  you  choose  hi  participate  you  can  change  your  mind 
at  any  time  and  withdraw  from  the  study  You  w  ill  not  be  penalized  in  any  way  or  lose  any  benefits  to 
which  you  w\vuld  otherwise  be  entitled  if  you  choose  not  to  participate  in  this  study  or  to  withdraw  The 
alternative  to  partKipating  in  this  research  is  to  not  participate  in  this  research 

Potential  Risks  and  Discomforts 

The  potential  risks  of  participating  «n  this  study  ate  minimal  You  will  be  sitting  in  front  ofa  large 
nvonnor  using  a  conventional  mouse  and  key  Kurd  to  interact  with  the  DaViTo  software.  The  eye  scan 
camera  systems  base  infrared  lights  associated  with  rhem  These  are  no  more  harmful  than  normal  room 
lighting  There  will  be  many  sale  guards  in  place  Id  protect  the  participant's  confidentiality:  however, 
there  is  always  a  minimal  nsk  that  (here  might  be  a  breach  of  confidentiality .  ||$  |  [| 
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Aatieipatvd  Bear  fils 

Anticipated  benefits  from  this  study  are  to  improved  the  usability  for  the  DaViTo  software,  and  provide 
visualization  data  in  a  clear  and  understandable  wav  fee  the  decision  maker  I  here  is  no  direct  benefit  to 
you  for  participating 

C  umpriivaliiia  fur  Participation 

No  tangible  compensation  will  be  given  so  the  participant 

Confidentiality  A.  Privacy  Art 

An y  information  tltat  is  obtained  during  this  study  will  be  kept  confidential  to  the  full  extent  permitted  by¬ 
law.  All  efforts,  within  reason,  wlH  be  made  to  keep  yoor  personal  information  in  your  research  record 
confidential  but  total  confidentiality  cannot  he  guaranteed 

All  surveys  will  be  protected  by  a  password  security  and  at  a  secure  NPS  terminal  If  it  is  necessary  to 
print  any  surveys,  all  personally  identifiable  information  (Pll)  will  be  scrubbed  so  the  participant  cannot 
he  identified  All  printed  surreys  will  be  stored  in  a  locked  drawer  in  a  secured  facility  when  not  in  use.  In 
the  final  publication,  no  names  will  be  used  fur  privacy  reasons  and  because  it  adds  no  value  to  the 
research. 

All  references  to  data  cotlccied  will  be  made  anonymous.  Your  name  will  be  encoded  as  a  participant 
number.  Only  principle  investigators  will  have  access  to  this  key  that  translates  an  identification  number 
to  your  nama. 

Points  of  C  ontact 

If  you  have  any  questions  or  comments  about  the  research,  or  you  experience  an  injury  or  have  questions 
about  any  discomforts  that  you  experience  while  taking  part  in  this  study  please  contact  the  Principal 
Investigator.  Dt.  Ji  llvuix.  831-656-3004.  ivanliifitps.edu  Questions  about  your  rights  us  a  research 
subject  or  any  other  concerns  may  be  addressed  to  the  Navy  Postgraduate  School  IRB  Chair,  CAPT  John 
Schmidt,  (8.1 1  )6>h-tM>t,  |k}c limiddnpy.edu. 

Statement  of  Consent 

I  have  read  the  information  prov  ided  above,  I  have  been  given  the  opportunity  to  ask  questions  and  all  the 
questions  have  been  answered  Id  my  satisfaction  I  have  been  ptov  ided  a  copy  of  this  form  for  tny  records 
and  I  agree  to  participate  m  this  study.  I  understand  that  by  agreeing  to  participate  m  this  research  and 
signmg  this  form.  I  do  not  waive  any  of  my  legal  rights. 


Participant's  Signature 


Date 


Researcher's  Signature 


I 'pdaicd  on:  4,'|0'I2 


Dale 


IPS  III 
APPRDfEO 

APR  1 1  200 
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DEMOGRAPHIC  SURVEY  ID*  (RANDOM  4  OMITS): _ 

1  Please  enter  your  age  in  yeors:  _ 

2  Please  indicate  your  military  service  branch  | Check  One) 

□  ARMY  n  NAVY  MARINE  CORPS  O  AIR  FORCE  RETIRED  MILITARY 

Li  INTIRNATIONAl.  MILITARY  MEMBER 

J  IF  English  your  primary  language?  (Checi  Orel  o  YES  i  NO 

4.  Please  indicate  your  military  grade  (e  g  0-2,  0-5, 0-4.  etc  I _ - _ 

5.  Please  indicate  <f  you  are  affiliated  with  the  development  o'  the  OaVITo  software  (Check  Onel 

□  YES  c  NO 

6  Are  you  now  or  have  you  ever  been  employed  as  an  analyst’  (Check  One)  a  YES  □  NO 

7  Have  you  ever  used  the  CXaViTo  software  before  you  participated  rt  this  study  (Crete  One) 

□  YES  oNO 

S.  Check  the  most  descriptive  unit  of  measurement  that  describes  the  amount  of  experience  you 
hove  with  computer  software: 

Q  None 

r  Novice  |word  processing,  emal,  web  surfing) 

Intermediate  user  (some  programming  in  at  least  one  python,  C,  java.  ruby.  |mp,  etc.) 
Expert  (Computer  Science  background,  or  can  create  complex  programs) 

9,  How  many  hours  per  week  do  you  spend  on  a  computer:  (Chert  One| 

O  S  hours  r  0-10  hours  o  11-15  hours  c  16-20  hours  □  21*-  hours 

10.  Do  you  have  20/20  vision,  or  correctable  to  20/20  vision?  Q  YES  u  NO 

11  Are  you  color  blind?  nYES  □  NO 

12.  Have  you  ever  used  ARC  GIS  Of  any  Other  geo  spatial  mapping  software:  ICbeck  One) 
n  YES  Ci  NO 

15  What  operating  system  do  you  prefer? 

L  Mac  C  Windows  Other 


ips  m 

ntflYfc 


limn 
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Quantitative  &  Subjective  Question* 


Fait  Survey 


Please  check  or  answer  the  following  question-;  about  the  experience  you  had  usirys  the  DaVito 
software  Please  use  the  following  scale  for  your  responses: 


Strongly  Qsagrec 

Neither 

Agree/Otsagrcc 

Strongly  Agree 

* 

4 

5 

Satisfaction  (refers  to:  perception,  feeling,  and  opinion): 

1.  I  feel  confident  in  my  a  battles  to  use  OaViTo  for  basic  analysis: 

0 1  u  2  a  3  rj  4  nS 

2.  I  think  I  can  successful  finish  basic  tasks  without  the  user's  manual 

ot  a2  □  3  c  4  aS 

3.  I  think  I  can  successfully  finish  compiei  tasks  wah  the  user’s  manual: 

□  1  li  2  o3  n  4  nS 

4  I  think  this  software  would  add  value  to  missions  that  I  have  performed: 

□  1  r  2  n  3  o  4  c  5 

5.  I  would  recommend  this  software  to  my  command: 

□  1  U 2  uS  c  4  n  5 

6.  I  feel  comfortable  using  OeVlTo  without  any  assistance: 

□  3  ti  2  n  3  o4  OS 

?,  Were  you  satisfied  with  the  performance  of  the  software? 

(Written  response) 


Visual  (refers  to:  representation  of  output): 

1  The  way  in  which  output  data  is  displayed  s  dear  to  mterpret 
Ol  c  2  03  04  u5 

2.  What  more  would  you  like  to  see  in  the  output  of  this  software? 

(Written  response) 

3.  Graphical  Representation  question  1 
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Quantitative  &  Subjective  Questions 


4.  Graphical  Representation  question  2 


5  Graphical  Representation  question  3 


6  Graphical  Representation  question  4 


ftcAi^vC  uv'k'*' 

coo  s-t  m  c-h  or ,  i  <  *  ;\ i  i»v lu  dd 

t 

•  lor 

"  \tn£ 

^cai^r  oi  Swvi  r  '«f 


Usefulness  (refers  to:  ability  to  achieve  coals,  ease  of  use,  and  wdUngnes*  to  usej: 

1.  The  tasks  were  easy  to  complete: 

01  □ 2  ui  c 4  a 5 

2-  What  was  the  most  difficult  task? 

(Written  Response  J 


3.  The  terminology  for  buttons  and  commands  was  clear  and  understandable: 

nl  c  2  al  04  t.-S 

4.  rf 1  were  an  analyst  I  would  use  this  tool  for  field  analysis: 

Dl  u  2  □  3  n«  IS 

5.  A  computer  novice  would  be  able  to  use  the  software  with  the  training  provided: 

ol  o2  c  3  j4  ui 

6.  A  computer  novice  would  be  able  to  use  this  software  wrth  utt  tho  user's  manual; 

□  1  n2  o  3  c 4  :j  5 

Efficiency  (refers  to:  quickness  to  accurately  accomplish  goal,  layout,  time  benchmark  and  narration): 

1 .  The  organisation  of  ihe  screen  matched  my  expectation  of  button  location 

L-l  n  2  □  3  n  4  ijS 

2  Sim4ar  icons  and  activities  are  consistently  placed: 

Ol  o2  03  Q4  tiS 
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QujiHit  olve  &  Subjective  Quest  ons 


3  Du*  to  my  experience  with  computer*.  navigation  of  DaVito  was  intuitive : 

□  1  a2  n  3  o<  c  5 

A  Was  the  visual  representation  easy  to  understand  ? 

(Written  Response) 


5.  The  color  scheme  for  the  software  was  appropriate: 

Ol  32  g3  u  4  a  S 

6.  The  tent  within  output  graphical  representations  was  easy  to  read  and  match  to  data: 

□  1  a  2  n3  C)4  oS 

7  The  software  header  bars  and  dropdown  lists  made  navigational  sense 
Ol  02  0 1  G 4  u5 

8.  The  software  was  stable,  no  freeing: 
u  1  n  2  n3  r  4  35 

Effectiveness  (refers  to:  software  behaves  as  expected,  workflow,  error  rate,  mouse  dkk  bench 
mark): 


1.  I  understood  where  I  was  at  in  the  software  as  I  progressed  through  a  task 

□  1  O  2  cl  04  at 

2.  The  mouse  interacted  with  the  software  as  I  expected  it  to: 

ol  o  2  n3  o4  c5 

3  Was  the  software  simple  to  use? 

(Written  response) 


4  I  was  satnhed  with  the  reaction  time  of  the  software: 

cl  ol  t  3  a4  r. 5 

5  I  felt  there  was  an  unnecessary  amount  of  mouse  clicking  to  complete  the  given  task : 

ol  ol  nl  u  4  oS 

6.  I  made  frequent  errors  trying  to  complete  the  given  task: 

□  1  nl  □  3  i-4  dS 
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Quantitative  K  Subjective  Questions 


learnabifcty  (refers  to:  competent*,  training  time,  confidence,  relearn  an  activity): 

1.  I  think  moo  peoplecwould  figure  out  how  to  us*  this  software  with  the  training  aids  provided: 

□  1  n  2  ol  at  OS 

J.  I  think  most  peopte  could  figure  out  how  to  use  this  software  with  just  the  user's  manual: 

cl  'Jl  o  1  d  *  a  S 

3.  No  more  training  «s  needed  to  effectively  use  this  software 

ol  u 2  c3  nt  as 

4.  If  given  these  tasks  in  one  month  to  complete  again.  I  feel  I  could  execute  them: 
ol  c  2  oi  u4  dS 

5.  I  could  show  someone  who  has  no  e>perience  with  OaViTo  basic  functionality 

Ol  n2  n 3  r  4  c  5 

improvements  | refer  to  the  users  suggestions): 

I  User* s  manual  is  easily  understandable 

□  1  02  u  3  at  n5 

2.  More  trnirwtg  aids  are  needed  for  me  to  effect ively  learn  to  use  the  software: 

□  1  u  2  o  3  o4  c  5 

3.  How  would  you  improve  OaViTo? 

(Written  response) 


4  What  was  the  most  frustrating  feature  of  OaViTo? 

{Written  response) 


S.  What  <t«d  you  like  most  about  OaViTo? 
(Written  response) 


6  What  wotdd  you  change  about  the  color  scheme? 

(Written  response) 


7.  Mow  would  you  Improve  the  moose  interact  on  with  the  software7 

(Written  res  panes) 
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Quantitative  &  Sub|ectrve  Questions 


B  Mow  would  you  Uke  to  tee  the  software  employed? 

{Written  response) 
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Graphics  Questions 


o 

Do  you  know  whit  this  icon  means? 

O  Drawing  O  Pan  ci  Distance  o  Navigation  u  Gestures  n  Go  forward  to  neat  projection 
a  Go  back  to  previous  protection  a  Map  Nawgation  c  •  don't  know 


A 


‘  Do  you  know  what  this  icon  means? 


□  Drawing  o  Pan  c  Distance  □  Navigation  d  Gestures  i:  Go  forward  to  neat  projection 
O  Go  back  to  previous  protection  O  Map  Navigation  c  I  don't  know 


- i  Do  you  know  what  this  icon  means? 

O  Drawing  □  Pan  a  Distance  1  Navigation  o  Gestures  r  Go  forward  to  next  protection 
D  Go  back  to  previous  projection  u  Map  Navigation  D  I  don't  know 


m> 

Do  you  know  what  this  icon  means? 

a  Drawing  o  Pan  o  Distance  o  Navigation  D  Gestures  c  Go  forward  to  next  projection 

D  Go  back  to  previous  protection  c  Map  Navigation  n  I  don't  know 


n  Drawing 
OGo  back  to 


you  know  what  this  Icon 
a  Pan  o  Distance 
previous  protection 


means? 

i  Navigation  d  Gestures 
u  Map  Navigation 


a  Go  forward  to  ne«1  projection 
a  I  don't  know 
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Graphics  Questions 


S] 


Do  you  know 
o  Drawing  q  Pan 
a  Go  bads  Id  previous 


what  this  con  means? 

u  Distance  c  Navigation  o  Gestures  □  Go  forward  to  neat  projection 
projection  c  Map  Navigation  □  I  don't  know 
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